Setting max_tokens: A Practical Guide
By Darus S Alfatah · 2025-09-25 · Optimasi · 6 menit
Author note: Our best value came from the 90th percentile of real answer lengths, plus a small margin.
Focus of this article
This article is specifically about setting the max_tokens parameter correctly, including how reasoning tokens affect reasoning models.
Read also: Prompt Tokens vs Completion Tokens: Why the Split Matters · Cutting AI Token Costs Without Hurting Quality
What the Parameter Does
max_tokens caps the length of the completion. It is not a target; the model still stops when it finishes. But a loose cap invites verbosity, and a tight cap truncates answers mid-sentence or produces invalid JSON.
Treat the value as a budget per feature, not a global setting.
Common Mistakes
First, assuming the model fills the limit. Second, setting it too low for structured output, where truncation breaks parsing. Third, forgetting that reasoning models spend tokens on internal steps, so the visible answer may be cut even when the cap looks generous.
Fourth, keeping one value for every feature; a classification task and a report generator have different needs.
Finding the Right Number
Collect thirty ideal answers from real usage, measure their token length, and take the ninetieth percentile. Add twenty percent margin for variation. Review quarterly as user behaviour and prompts change.
For structured output, measure the longest valid JSON your schema permits and set the cap slightly above it.
Combining With Instructions
A cap alone does not make answers concise. Add an explicit instruction such as “answer in at most five sentences”. Together, instruction and cap produce predictable length and cost.
Instructions also help when the model would otherwise add unnecessary preambles.
Cost Impact
Output tokens are usually the more expensive half, so caps directly affect the bill. But capping too tightly causes retries and manual corrections, which cost more than the saved tokens.
Measure quality and cost together before and after changing a cap.
Frequently Asked Questions
Does max_tokens include reasoning tokens? Sometimes; check your provider's documentation.
What happens when the limit is hit? The answer is cut and a finish reason indicates truncation.
Should I set it high and trim later? No, that increases cost and latency.
Can I change it per request? Yes, and you should for different features.
How do I detect truncation? Check the finish reason rather than guessing from length.
Conclusion
Set limits from real data, add explicit length instructions, and monitor truncation. Small, deliberate numbers beat one large default.
Implementing Per Feature
Keep a configuration map: classification 100, FAQ 300, summary 500, analysis 1,200, agent steps 2,000. Having numbers written down prevents accidental expansion when someone adds a feature quickly.
Review the map quarterly with real usage data. Limits that once looked generous may now truncate answers, and limits that were tight may be wasting quality.
Handling Truncation Gracefully
When the finish reason indicates truncation, retry once with a higher limit only if the content is critical; otherwise show the partial answer with a note. Silent truncation is the worst outcome, because users assume the answer was complete.
Keep the values in code review, not in a wiki, so changes are visible to the team.
Documenting the Why
Add a one-line comment beside each limit explaining how it was chosen. Future maintainers then understand which numbers came from measurement and which were guesses, and they can update the right ones.
An example of a truncated answer: If output stops mid-sentence or the JSON is incomplete, max_tokens is likely too small for that task.
Frequently Asked Questions
What is the optimization goal here?
This article covers reducing wasted tokens and cost without hurting output quality, making API usage more efficient.
Does this optimization lower answer quality?
No, when applied correctly. The focus is prompt and token efficiency, not reduced quality.
Where can I see my token usage?
In the BankAI dashboard, with per-model and per-period breakdowns.