On-Device AI: Small Models Now Run on Laptops and Phones
By Darus S Alfatah · 2026-10-20 · Berita AI · 10 menit
Author note: On-device wins for light, sensitive tasks; heavy tasks are still better on the server.
AI That No Longer Needs a Data Center
For years, running a capable language model meant renting GPUs in the cloud. A new trend reverses that assumption: chipmakers ship NPUs (Neural Processing Units) in laptops, phones, and even small devices, while 1–8 billion parameter models became good enough for everyday tasks.
As a result, many functions now run entirely locally: writing correction, document summarisation, speech transcription, semantic search in personal files, and assistants that understand device context. No internet, no per-token cost, and no data leaving the device.
Why It Matters
- Privacy — documents, recordings, and notes stay on the device.
- Latency — no network round trip, so short tasks feel instant.
- Cost — no token bill; cost is part of the hardware.
- Reliability — works on poor connections, common in many parts of Indonesia.
Honest Limits
Small local models cannot match large ones on complex reasoning, very long context, and up-to-date knowledge. Quality also depends on quantisation: 4-bit models save memory but can lose precision on code or maths.
Memory is a hard constraint. An 8B model with long context can consume over ten gigabytes, which is too much for an 8 GB laptop. Cooling and battery life are affected during long inference.
A Realistic Hybrid Architecture
- Local first for routine and sensitive tasks: meeting summaries, document classification, autocomplete.
- Cloud when needed for heavy reasoning, multimodality, and fresh knowledge.
- Router in the app decides based on task type and difficulty.
This pattern cuts API cost significantly while preserving data compliance — highly relevant to Indonesia’s data protection law.
Tooling for Developers
The local ecosystem is maturing: runtimes such as Ollama and llama.cpp make running GGUF models easy, platform SDKs expose NPU APIs, and many open models ship in device-optimised variants.
For a quick experiment, start with a 3B–4B model for classification and summarisation, then measure quality on your data. If it is sufficient, you do not need an API call for that feature at all.
Starting With Clear Wins
Choose functions where data is sensitive and answers are short: summarising notes, correcting spelling, classifying documents. Avoid starting with a general-purpose chatbot, because small models disappoint on broad knowledge and users conclude the AI is bad.
Managing Devices Without an NPU
if (hasNPU) use 4-bit model
else if (ram >= 8GB) use smaller 4-bit model, short context
else send to server (with user consent)
Always give users a switch. Local AI should be an option they understand, not a hidden mode that drains their battery.
Measure Before and After
Before shipping on-device features, record baseline metrics: response time, accuracy, and complaints on the server version. After the local feature ships, compare the same metrics. If speed improves but accuracy drops, split the work: local model for drafts, server model for final verification.
Conclusion
On-device AI is not a cloud replacement but a new layer in application architecture. Developers who assume AI always runs on a server miss a significant opportunity: instant, private, and cheap experiences without heavy infrastructure.
Sources & References
An example of an on-device task: Spell-check, autocomplete, and short summaries run well locally. Heavy tasks like long analysis stay on the server.
Frequently Asked Questions
Does this news stay relevant over time?
AI moves fast; this article gives context and principles that remain useful, and is updated when important developments happen.
How do I keep up with the latest AI news?
Follow the BankAI blog and the model catalog on the AI Models page, updated with available models.
Do these developments affect developers directly?
Many affect model choice, cost, and integration — all covered from a practical angle.