Behind AI: The GPU, HBM, and Energy Race

By Darus S Alfatah · 2026-10-14 · Infrastruktur · 11 menit

Behind AI: The GPU, HBM, and Energy Race

Author note: GPU and energy cost are now part of the equation, not just model price per token.

AI as an Infrastructure Load

Every AI answer is the result of computation in a data center. When millions of people use models simultaneously, demand for accelerators, high-bandwidth memory, networking, and electricity surges. That is why AI pricing depends not only on research but on the hardware supply chain.

Understanding this layer helps developers make realistic decisions: why some models cost more, why latency changes at peak hours, and why capacity can be limited in a region.

GPUs and HBM

Large models need fast memory in large quantities. HBM (High Bandwidth Memory) is the critical component: capacity determines which models fit, and bandwidth determines inference speed. HBM production is concentrated among few suppliers, making it a supply bottleneck.

The effect is direct: providers compete for allocation, procurement lead times stretch, and hardware depreciation must be recovered through per-token pricing.

Energy and Cooling

AI data centers draw far more power than conventional ones. The International Energy Agency projects significant growth in data center electricity use as AI adoption expands, while stressing efficiency and clean energy sources.

Cooling is a major part: high rack density demands liquid cooling, which changes building design and site selection. Countries with cheap power, favourable climate, and friendly regulation attract investment.

Implications for Indonesia and Southeast Asia

Several global companies have announced data center investments in Southeast Asia, including Indonesia. The long-term effect: lower latency for local users and the potential to process AI domestically — important for data compliance.

However, building data centers does not automatically make the best models available locally. Often the capacity serves specific services while frontier models are still served from other regions.

Strategies for Efficient Developers

  • Use the smallest sufficient model — saves GPU time and cost.
  • Use caching — identical long system prompts should not be recomputed.
  • Batch non-real-time work — overnight processing is often cheaper.
  • Monitor latency by region — choose the nearest endpoint for interactive apps.
  • Design for degradation — fall back to a smaller model when capacity is tight.

Sustainability

Criticism of AI energy use is valid. Industry responses include hardware efficiency, renewable energy, and model optimisation. For developers, efficiency is also responsibility: every unnecessary prompt is unnecessary computation.

Handling Capacity Uncertainty

Design for change: provide equivalent models, queue non-urgent work, and tell users when the system is busy. Teams with fallbacks stay calm when a provider has an outage or tightens quotas. Set daily cost caps and automatic alerts so traffic spikes do not create surprise bills.

Conclusion

AI is a physical technology, not magic. Chip, memory, and power availability define what is possible. Developers who understand this layer choose models, architectures, and cost expectations more realistically.

Sources & References

What Developers Can Control

You cannot change GPU prices, but you can change how often models are called. Measure requests you do not actually need: autocomplete firing on every keystroke, summaries regenerated on every page load, retries without limits. Fixing these often reduces cost more than negotiating with a provider.

An example of the trade-off: Renting your own GPU can be cheap at high volume, but expensive when idle. For spiky traffic, an API per token is often cheaper.

Frequently Asked Questions

What matters when choosing AI infrastructure?

Cost, latency, scalability, and provider reliability. Using a gateway like BankAI abstracts many providers behind one API key.

Do I need my own GPU to use AI?

Not necessarily. You can use an API from a provider without owning a GPU; own GPUs matter for special/high-privacy needs.

How do I keep an AI-dependent app reliable?

Provide fallback to another model, handle timeouts, and retry in a measured way.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing