Setting max_tokens: A Practical Guide

By Darus S Alfatah · 2025-09-25 · Optimasi · 6 menit

Setting max_tokens: A Practical Guide

Author note: Our best value came from the 90th percentile of real answer lengths, plus a small margin.

Focus of this article

This article is specifically about setting the max_tokens parameter correctly, including how reasoning tokens affect reasoning models.

Read also: Prompt Tokens vs Completion Tokens: Why the Split Matters · Cutting AI Token Costs Without Hurting Quality

What the Parameter Does

max_tokens caps the length of the completion. It is not a target; the model still stops when it finishes. But a loose cap invites verbosity, and a tight cap truncates answers mid-sentence or produces invalid JSON.

Treat the value as a budget per feature, not a global setting.

Common Mistakes

First, assuming the model fills the limit. Second, setting it too low for structured output, where truncation breaks parsing. Third, forgetting that reasoning models spend tokens on internal steps, so the visible answer may be cut even when the cap looks generous.

Fourth, keeping one value for every feature; a classification task and a report generator have different needs.

Finding the Right Number

Collect thirty ideal answers from real usage, measure their token length, and take the ninetieth percentile. Add twenty percent margin for variation. Review quarterly as user behaviour and prompts change.

For structured output, measure the longest valid JSON your schema permits and set the cap slightly above it.

Combining With Instructions

A cap alone does not make answers concise. Add an explicit instruction such as “answer in at most five sentences”. Together, instruction and cap produce predictable length and cost.

Instructions also help when the model would otherwise add unnecessary preambles.

Cost Impact

Output tokens are usually the more expensive half, so caps directly affect the bill. But capping too tightly causes retries and manual corrections, which cost more than the saved tokens.

Measure quality and cost together before and after changing a cap.

Frequently Asked Questions

Does max_tokens include reasoning tokens? Sometimes; check your provider's documentation.

What happens when the limit is hit? The answer is cut and a finish reason indicates truncation.

Should I set it high and trim later? No, that increases cost and latency.

Can I change it per request? Yes, and you should for different features.

How do I detect truncation? Check the finish reason rather than guessing from length.

Conclusion

Set limits from real data, add explicit length instructions, and monitor truncation. Small, deliberate numbers beat one large default.

Implementing Per Feature

Keep a configuration map: classification 100, FAQ 300, summary 500, analysis 1,200, agent steps 2,000. Having numbers written down prevents accidental expansion when someone adds a feature quickly.

Review the map quarterly with real usage data. Limits that once looked generous may now truncate answers, and limits that were tight may be wasting quality.

Handling Truncation Gracefully

When the finish reason indicates truncation, retry once with a higher limit only if the content is critical; otherwise show the partial answer with a note. Silent truncation is the worst outcome, because users assume the answer was complete.

Keep the values in code review, not in a wiki, so changes are visible to the team.

Documenting the Why

Add a one-line comment beside each limit explaining how it was chosen. Future maintainers then understand which numbers came from measurement and which were guesses, and they can update the right ones.

An example of a truncated answer: If output stops mid-sentence or the JSON is incomplete, max_tokens is likely too small for that task.

Frequently Asked Questions

What is the optimization goal here?

This article covers reducing wasted tokens and cost without hurting output quality, making API usage more efficient.

Does this optimization lower answer quality?

No, when applied correctly. The focus is prompt and token efficiency, not reduced quality.

Where can I see my token usage?

In the BankAI dashboard, with per-model and per-period breakdowns.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

Related articles

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing