How to Benchmark AI Models Yourself Before Production

By Darus S Alfatah · 2026-11-08 · Optimasi · 9 menit

How to Benchmark AI Models Yourself Before Production

Author note: A public leaderboard once led us to a model that lost in our own domain. Since then, our own test set has been mandatory.

Why Public Benchmarks Are Not Enough

AI model leaderboards are tempting but often misleading. A high score on a general maths exam does not mean the model is good at extracting invoices or answering your customers. Differences in domain, language, and format make public benchmark results unrepresentative of real cases. That is why your own benchmark is the most reliable way to choose a model.

Step 1: Collect Real Tasks

Take 30–100 examples from actual cases, including hard and easy ones. For each, define the correct answer or clear success criteria. This collection is the test set and becomes the fixed reference for all comparisons.

Step 2: Choose Suitable Metrics

  • Accuracy/deterministic — for classification and extraction (right/wrong).
  • Rubric score — for open answers (1–5 on several aspects).
  • Format adherence — does the output obey the schema.
  • Latency — time to first response and total.
  • Cost — tokens per completed task.

Step 3: Run It Fairly

Use the same prompt for all models. Do not tune a prompt for one model without applying it to all. Run several times because output can vary, then average. Record cost and latency alongside quality.

for each model:
  for each example in test_set:
    answer = model(same_prompt)
    score = rubric(answer, reference)
  compute average score, latency, tokens

Step 4: Choose by Completed Task

The cheapest model per token is not necessarily the cheapest per task if it needs many attempts. Compute the cost of completing one task correctly. Often a mid-tier model is more economical than the cheapest one.

Evaluating with a Model as Judge

For open answers, another model can score quality against a rubric. Convenient, but biased — models favour their own output. Calibrate with human scoring on a subset, and do not fully replace humans for important decisions.

Testing Edge Cases

Include unusual examples: empty input, mixed language, long requests, and misleading prompts. Robustness on edge cases is often more important than the average score.

Common Mistakes

  • Testing on examples also used to build the prompt (leakage).
  • Vague metrics without success criteria.
  • Ignoring cost and latency.
  • Too few examples, so results are unstable.

Keeping the Test Set Relevant

The world changes; cases relevant a year ago may be stale. Review the test set quarterly, add new examples from production, and drop those no longer representative. Keep the test set separate from the data used to tune prompts to avoid leakage.

Automating Evaluation

Make evaluation part of the release pipeline. Whenever the integration changes, run the test set and compare against the baseline. Quality regressions are then caught before reaching users.

FAQ

How many examples for an ideal test set?

At least 30; 100 or more gives more stable results.

Can I use public benchmarks?

Yes as a starting point, but base the final decision on your own data.

Is a model-as-judge enough?

Convenient but biased. Calibrate with human scoring on a subset.

How often to run evaluation?

On every integration change, and review the test set quarterly.

What else to measure besides quality?

Cost per completed task and latency, since both determine production viability.

Key Takeaways

  • Public benchmarks do not represent your domain.
  • Build a test set from real cases with clear references.
  • Run all models with the same prompt.
  • Compare cost per completed task, not per token.
  • Test edge cases and calibrate human scoring.

Conclusion

Your own benchmark is a small investment with a big payoff: a model decision based on data, not trends. Start with 30 examples, measure honestly, and review periodically. With one API key at BankAI, you can compare many models from the same code without changing your integration.

Example: 30 tasks, four criteria: Suppose you are choosing between three models. Take 30 real tasks from your own work, run them on all three, then score 1–5 for: correct, complete, on-format, and reasonable on cost. The best model is the one that wins on your tasks — not the one that is newest in the news.

Frequently Asked Questions

What is the optimization goal here?

This article covers reducing wasted tokens and cost without hurting output quality, making API usage more efficient.

Does this optimization lower answer quality?

No, when applied correctly. The focus is prompt and token efficiency, not reduced quality.

Where can I see my token usage?

In the BankAI dashboard, with per-model and per-period breakdowns.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

Related articles

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing