Tokens, Context Windows, and Temperature Explained

By Darus S Alfatah · 2025-07-26 · Edukasi · 8 menit

Tokens, Context Windows, and Temperature Explained

Author note: Temperature 0 is more repeatable for data tasks; for creative work, higher values help variation.

Tokens: The Unit of Billing

Models do not read words; they read tokens, which are fragments of words. In Indonesian, one word often becomes two or three tokens because of prefixes and suffixes, so a sentence costs more than the same idea in English. That is why estimating cost by counting words tends to underestimate the bill.

Both input and output are billed. Long prompts, long histories, and long documents all add up, and they add up on every single request.

Context Window: The Hard Limit

The context window is the maximum number of tokens a model can consider at once, including the prompt and the answer. If you send a 100,000-token document to a 128,000-token model, only 28,000 tokens remain for the reply. Exceeding the limit either fails or truncates, depending on the provider.

A large window is convenient but not free: you pay for what you send, and accuracy can degrade when irrelevant material is included. Retrieval usually beats brute force.

Temperature: Randomness Control

Temperature controls how much variation the model introduces. Near zero, answers become consistent and repeatable — ideal for extraction, classification, and anything that must be stable. Around 0.7, output is varied but coherent — good for brainstorming and drafting. Above one, creativity rises and coherence falls.

extraction  → temperature 0
summarising → temperature 0.2
marketing   → temperature 0.7–0.9

How They Interact

These settings are not independent. A long context with high temperature produces more unpredictable output. A short context with low temperature produces reliable but possibly repetitive answers. Tune them together and record the combination per feature so results are reproducible.

Measuring Before Launching

Before shipping, measure average tokens from fifty real examples and set input and output limits. Without limits, one unusual request can consume a day's budget.

cost = (tokens_in * price_in + tokens_out * price_out) / 1_000_000

Common Configuration Errors

  • High temperature for tasks that need consistency.
  • Sending a long history when the question concerns one paragraph.
  • Assuming a large context window is free.
  • Never testing behaviour when the context is truncated.

Frequently Asked Questions

How many tokens is a page of text? Roughly 500–800 for Indonesian, depending on vocabulary and formatting.

Does a bigger context window improve quality? Only when the extra material is relevant. Otherwise it adds cost and noise.

Can I reduce tokens without losing meaning? Yes: remove filler, use bullet points, and retrieve only relevant sections.

Is temperature the same as top-p? They are related sampling controls; adjust one at a time.

Where do I monitor usage? In your provider dashboard per model, and internally per feature.

Conclusion

Tokens decide cost, context decides what is possible, and temperature decides behaviour. Understanding all three turns guesswork into deliberate configuration.

Practical Calibration

Pick one feature and run ten real inputs through three configurations: low temperature with short context, medium temperature with retrieved context, and high temperature with full context. Record cost, latency, and a blind quality score from a colleague. You will usually find that the middle configuration wins, because relevance beats volume and stability beats variety for most business tasks.

Repeat this exercise whenever you switch models. Tokenisers differ, so the same text can cost more or less, and a prompt that filled a tenth of the window on one model may fill a fifth on another.

Example task settings: For extracting data from documents, set a low temperature for consistency. For generating slogans, set it higher for variety.

Frequently Asked Questions

What is the core topic of this article?

This article explains the fundamentals of AI APIs and gateways in plain language for developers, with practical examples.

Do I need a technical background?

No. The material builds up from the basics, so beginners can follow while experienced developers can jump to the relevant section.

How do I get started with an AI API after reading this?

Create a BankAI account, get an API key, then change your Base URL to BankAI’s OpenAI-compatible endpoint.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

Related articles

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing