Tokens, Context Windows, and Temperature Explained
By Darus S Alfatah · 2025-07-26 · Edukasi · 8 menit
Author note: Temperature 0 is more repeatable for data tasks; for creative work, higher values help variation.
Tokens: The Unit of Billing
Models do not read words; they read tokens, which are fragments of words. In Indonesian, one word often becomes two or three tokens because of prefixes and suffixes, so a sentence costs more than the same idea in English. That is why estimating cost by counting words tends to underestimate the bill.
Both input and output are billed. Long prompts, long histories, and long documents all add up, and they add up on every single request.
Context Window: The Hard Limit
The context window is the maximum number of tokens a model can consider at once, including the prompt and the answer. If you send a 100,000-token document to a 128,000-token model, only 28,000 tokens remain for the reply. Exceeding the limit either fails or truncates, depending on the provider.
A large window is convenient but not free: you pay for what you send, and accuracy can degrade when irrelevant material is included. Retrieval usually beats brute force.
Temperature: Randomness Control
Temperature controls how much variation the model introduces. Near zero, answers become consistent and repeatable — ideal for extraction, classification, and anything that must be stable. Around 0.7, output is varied but coherent — good for brainstorming and drafting. Above one, creativity rises and coherence falls.
extraction → temperature 0
summarising → temperature 0.2
marketing → temperature 0.7–0.9
How They Interact
These settings are not independent. A long context with high temperature produces more unpredictable output. A short context with low temperature produces reliable but possibly repetitive answers. Tune them together and record the combination per feature so results are reproducible.
Measuring Before Launching
Before shipping, measure average tokens from fifty real examples and set input and output limits. Without limits, one unusual request can consume a day's budget.
cost = (tokens_in * price_in + tokens_out * price_out) / 1_000_000
Common Configuration Errors
- High temperature for tasks that need consistency.
- Sending a long history when the question concerns one paragraph.
- Assuming a large context window is free.
- Never testing behaviour when the context is truncated.
Frequently Asked Questions
How many tokens is a page of text? Roughly 500–800 for Indonesian, depending on vocabulary and formatting.
Does a bigger context window improve quality? Only when the extra material is relevant. Otherwise it adds cost and noise.
Can I reduce tokens without losing meaning? Yes: remove filler, use bullet points, and retrieve only relevant sections.
Is temperature the same as top-p? They are related sampling controls; adjust one at a time.
Where do I monitor usage? In your provider dashboard per model, and internally per feature.
Conclusion
Tokens decide cost, context decides what is possible, and temperature decides behaviour. Understanding all three turns guesswork into deliberate configuration.
Practical Calibration
Pick one feature and run ten real inputs through three configurations: low temperature with short context, medium temperature with retrieved context, and high temperature with full context. Record cost, latency, and a blind quality score from a colleague. You will usually find that the middle configuration wins, because relevance beats volume and stability beats variety for most business tasks.
Repeat this exercise whenever you switch models. Tokenisers differ, so the same text can cost more or less, and a prompt that filled a tenth of the window on one model may fill a fifth on another.
Example task settings: For extracting data from documents, set a low temperature for consistency. For generating slogans, set it higher for variety.
Frequently Asked Questions
What is the core topic of this article?
This article explains the fundamentals of AI APIs and gateways in plain language for developers, with practical examples.
Do I need a technical background?
No. The material builds up from the basics, so beginners can follow while experienced developers can jump to the relevant section.
How do I get started with an AI API after reading this?
Create a BankAI account, get an API key, then change your Base URL to BankAI’s OpenAI-compatible endpoint.