Why AI Token Prices Keep Falling: Quantisation, MoE, and Distillation

By Darus S Alfatah · 2026-10-21 · Berita AI · 11 menit

Why AI Token Prices Keep Falling: Quantisation, MoE, and Distillation

Author note: Quantisation makes large models cheaper, but quality on hard tasks still needs retesting.

Why AI Gets Cheaper Fast

Prices per million tokens for mid-tier models have fallen dramatically compared with two years ago. The cause is not one breakthrough but a combination of hardware efficiency, model compression, and architectures that use less compute. For API users the effect is immediate: features that were too expensive are now viable in production.

This article covers four levers: quantisation, Mixture of Experts, distillation, and batching/caching — and what they mean for your technical decisions.

1. Quantisation

Quantisation reduces weight precision, for example from 16-bit to 8-bit or 4-bit. Memory needs fall two to four times, throughput rises, and GPU cost per token drops. In many cases quality loss is small and can be offset by calibration.

In practice 8-bit is almost always safe for general tasks. 4-bit saves more but is more sensitive on long reasoning and complex code, so test with your own data.

2. Mixture of Experts (MoE)

An MoE model has many experts but activates only a fraction per token. Total parameters can be huge while inference cost stays proportional to active parameters. That is why some models feel as capable as much larger ones at lower cost.

The trade-off: MoE needs more memory to hold all experts, making it better suited to providers serving many users than to a personal laptop.

3. Distillation

Distillation trains a small model to imitate a large one on a specific data distribution. The student is not as good generally, but often matches on narrow tasks — at a fraction of the cost. Many small models in circulation today are distilled from large reasoning models.

For business applications this is highly useful: if ninety percent of your requests are ticket classification, a small distilled model can cut cost by over eighty percent.

4. Batching, Caching, and Routing

  • Batching — processing many requests together improves GPU utilisation.
  • KV/prompt caching — avoids recomputing identical context; huge savings for long system prompts.
  • Routing — send easy requests to small models and hard ones to large models.
  • Speculative decoding — a small model proposes tokens and a large one verifies, cutting latency.

What It Means for Your App

Falling prices are not a reason to be wasteful. Design mistakes — overly long prompts, excessive context, oversized models for trivial tasks — now separate cost-healthy apps from the rest.

Three habits matter most: measure tokens per feature rather than in aggregate, trim context aggressively, and cache repeated answers.

Measuring the Trade-off

mode   | accuracy | latency | cost
fp16   | 92%      | 1.8 s   | 100%
int8   | 91%      | 1.2 s   | 62%
int4   | 88%      | 0.9 s   | 41%

These illustrative numbers show the usual pattern: small loss at 8-bit, larger at 4-bit. For tolerant tasks 4-bit may work; for data extraction 8-bit is safer.

Conclusion

Inference efficiency is why AI became affordable for Indonesian SMEs. Understanding the techniques helps you choose models and architectures deliberately, rather than following every release.

Sources & References

An example of quality retest: A quantised model is cheaper and faster, but rescore it on your ten hardest examples to confirm quality is intact.

Frequently Asked Questions

Does this news stay relevant over time?

AI moves fast; this article gives context and principles that remain useful, and is updated when important developments happen.

How do I keep up with the latest AI news?

Follow the BankAI blog and the model catalog on the AI Models page, updated with available models.

Do these developments affect developers directly?

Many affect model choice, cost, and integration — all covered from a practical angle.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

Related articles

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing