Behind AI: The GPU, HBM, and Energy Race
By Darus S Alfatah · 2026-10-14 · Infrastruktur · 11 menit
Author note: GPU and energy cost are now part of the equation, not just model price per token.
AI as an Infrastructure Load
Every AI answer is the result of computation in a data center. When millions of people use models simultaneously, demand for accelerators, high-bandwidth memory, networking, and electricity surges. That is why AI pricing depends not only on research but on the hardware supply chain.
Understanding this layer helps developers make realistic decisions: why some models cost more, why latency changes at peak hours, and why capacity can be limited in a region.
GPUs and HBM
Large models need fast memory in large quantities. HBM (High Bandwidth Memory) is the critical component: capacity determines which models fit, and bandwidth determines inference speed. HBM production is concentrated among few suppliers, making it a supply bottleneck.
The effect is direct: providers compete for allocation, procurement lead times stretch, and hardware depreciation must be recovered through per-token pricing.
Energy and Cooling
AI data centers draw far more power than conventional ones. The International Energy Agency projects significant growth in data center electricity use as AI adoption expands, while stressing efficiency and clean energy sources.
Cooling is a major part: high rack density demands liquid cooling, which changes building design and site selection. Countries with cheap power, favourable climate, and friendly regulation attract investment.
Implications for Indonesia and Southeast Asia
Several global companies have announced data center investments in Southeast Asia, including Indonesia. The long-term effect: lower latency for local users and the potential to process AI domestically — important for data compliance.
However, building data centers does not automatically make the best models available locally. Often the capacity serves specific services while frontier models are still served from other regions.
Strategies for Efficient Developers
- Use the smallest sufficient model — saves GPU time and cost.
- Use caching — identical long system prompts should not be recomputed.
- Batch non-real-time work — overnight processing is often cheaper.
- Monitor latency by region — choose the nearest endpoint for interactive apps.
- Design for degradation — fall back to a smaller model when capacity is tight.
Sustainability
Criticism of AI energy use is valid. Industry responses include hardware efficiency, renewable energy, and model optimisation. For developers, efficiency is also responsibility: every unnecessary prompt is unnecessary computation.
Handling Capacity Uncertainty
Design for change: provide equivalent models, queue non-urgent work, and tell users when the system is busy. Teams with fallbacks stay calm when a provider has an outage or tightens quotas. Set daily cost caps and automatic alerts so traffic spikes do not create surprise bills.
Conclusion
AI is a physical technology, not magic. Chip, memory, and power availability define what is possible. Developers who understand this layer choose models, architectures, and cost expectations more realistically.
Sources & References
What Developers Can Control
You cannot change GPU prices, but you can change how often models are called. Measure requests you do not actually need: autocomplete firing on every keystroke, summaries regenerated on every page load, retries without limits. Fixing these often reduces cost more than negotiating with a provider.
An example of the trade-off: Renting your own GPU can be cheap at high volume, but expensive when idle. For spiky traffic, an API per token is often cheaper.
Frequently Asked Questions
What matters when choosing AI infrastructure?
Cost, latency, scalability, and provider reliability. Using a gateway like BankAI abstracts many providers behind one API key.
Do I need my own GPU to use AI?
Not necessarily. You can use an API from a provider without owning a GPU; own GPUs matter for special/high-privacy needs.
How do I keep an AI-dependent app reliable?
Provide fallback to another model, handle timeouts, and retry in a measured way.