Why AI Token Prices Keep Falling: Quantisation, MoE, and Distillation
By Darus S Alfatah · 2026-10-21 · Berita AI · 11 menit
Author note: Quantisation makes large models cheaper, but quality on hard tasks still needs retesting.
Why AI Gets Cheaper Fast
Prices per million tokens for mid-tier models have fallen dramatically compared with two years ago. The cause is not one breakthrough but a combination of hardware efficiency, model compression, and architectures that use less compute. For API users the effect is immediate: features that were too expensive are now viable in production.
This article covers four levers: quantisation, Mixture of Experts, distillation, and batching/caching — and what they mean for your technical decisions.
1. Quantisation
Quantisation reduces weight precision, for example from 16-bit to 8-bit or 4-bit. Memory needs fall two to four times, throughput rises, and GPU cost per token drops. In many cases quality loss is small and can be offset by calibration.
In practice 8-bit is almost always safe for general tasks. 4-bit saves more but is more sensitive on long reasoning and complex code, so test with your own data.
2. Mixture of Experts (MoE)
An MoE model has many experts but activates only a fraction per token. Total parameters can be huge while inference cost stays proportional to active parameters. That is why some models feel as capable as much larger ones at lower cost.
The trade-off: MoE needs more memory to hold all experts, making it better suited to providers serving many users than to a personal laptop.
3. Distillation
Distillation trains a small model to imitate a large one on a specific data distribution. The student is not as good generally, but often matches on narrow tasks — at a fraction of the cost. Many small models in circulation today are distilled from large reasoning models.
For business applications this is highly useful: if ninety percent of your requests are ticket classification, a small distilled model can cut cost by over eighty percent.
4. Batching, Caching, and Routing
- Batching — processing many requests together improves GPU utilisation.
- KV/prompt caching — avoids recomputing identical context; huge savings for long system prompts.
- Routing — send easy requests to small models and hard ones to large models.
- Speculative decoding — a small model proposes tokens and a large one verifies, cutting latency.
What It Means for Your App
Falling prices are not a reason to be wasteful. Design mistakes — overly long prompts, excessive context, oversized models for trivial tasks — now separate cost-healthy apps from the rest.
Three habits matter most: measure tokens per feature rather than in aggregate, trim context aggressively, and cache repeated answers.
Measuring the Trade-off
mode | accuracy | latency | cost
fp16 | 92% | 1.8 s | 100%
int8 | 91% | 1.2 s | 62%
int4 | 88% | 0.9 s | 41%
These illustrative numbers show the usual pattern: small loss at 8-bit, larger at 4-bit. For tolerant tasks 4-bit may work; for data extraction 8-bit is safer.
Conclusion
Inference efficiency is why AI became affordable for Indonesian SMEs. Understanding the techniques helps you choose models and architectures deliberately, rather than following every release.
Sources & References
An example of quality retest: A quantised model is cheaper and faster, but rescore it on your ten hardest examples to confirm quality is intact.
Frequently Asked Questions
Does this news stay relevant over time?
AI moves fast; this article gives context and principles that remain useful, and is updated when important developments happen.
How do I keep up with the latest AI news?
Follow the BankAI blog and the model catalog on the AI Models page, updated with available models.
Do these developments affect developers directly?
Many affect model choice, cost, and integration — all covered from a practical angle.