How to Measure AI Output Quality: Evals, Rubrics, and LLM-as-Judge

By Darus S Alfatah · 2026-10-31 · Teknis · 12 menit

How to Measure AI Output Quality: Evals, Rubrics, and LLM-as-Judge

Author note: We score output with a rubric, not taste. Consistent scores beat impressions.

The Problem: Everything Feels Better After a Prompt Change

Teams often change prompts based on impressions. After the change, a few examples feel better, and the change is declared a success. The problem: the examples are usually five to ten hand-picked cases. Regressions elsewhere go unnoticed.

Evaluation turns this from impression into evidence. You do not need expensive tooling; a fixed test set and a simple rubric are enough for most teams.

Step 1: Build a Test Set

Take 30–50 real cases that represent everyday usage, including hard and edge cases. For each, write the ideal answer or pass criteria. Keep it in a readable file such as JSON or a spreadsheet.

{
  "id": "faq-01",
  "input": "How long does QRIS top-up take?",
  "must_include": ["automatic", "a few minutes"],
  "forbidden": ["credit card"],
  "format": "max 3 sentences"
}

Step 2: Define a Rubric

A good rubric has few dimensions and clear definitions. Example for answers: correct (no factual errors), complete (answers the question), format-compliant, and safe (no data leaks). Score 0–2 per dimension, then sum.

  • 0 — wrong or rule-breaking.
  • 1 — partly right, needs fixes.
  • 2 — meets all criteria.

Step 3: Run Automated Checks

Anything rule-based should be checked by code: required keywords present, forbidden words absent, maximum length, valid JSON. It is cheap, fast, and unbiased.

function check(answer, testCase) {
  const failures = [];
  for (const word of testCase.must_include || [])
    if (!answer.toLowerCase().includes(word.toLowerCase()))
      failures.push('missing: ' + word);
  for (const word of testCase.forbidden || [])
    if (answer.toLowerCase().includes(word.toLowerCase()))
      failures.push('forbidden: ' + word);
  if (testCase.format === 'max 3 sentences' && answer.split('.').length > 4)
    failures.push('too long');
  return failures;
}

Step 4: LLM-as-Judge for Subjective Aspects

For aspects that cannot be rule-checked — tone, clarity, completeness — another model can act as judge. Give it the same rubric and ask for structured output.

Score the answer below from 0-2 on: correct, complete, clear.
Return JSON: {"correct":0-2,"complete":0-2,"clear":0-2,"reason":"..."}
Do not score writing style outside the rubric.

Restrict the judge to defined aspects. Without a tight rubric, judges tend to reward longer answers.

Step 5: Look at Numbers, Not Stories

Run the test set whenever the prompt or model changes. Store results so they can be compared. If a specific dimension drops, you know exactly what to fix instead of changing everything at once.

Common Mistakes

  • Tests that are too easy — everything passes, nothing is learned.
  • Too many dimensions — hard to interpret and inconsistent.
  • No prompt versioning — results cannot be reproduced.
  • Relying only on a model judge — bias goes undetected.

Running Evals Regularly

Schedule evals on every prompt, model, or data-source change. Automate them in the pipeline so results are stored and comparable across versions. Review ten samples manually each month: scores can stay flat while quality quietly declines, and only human reading catches tone or subtle oddities.

Avoiding Self-Deception

Never evaluate on cases used to build the prompt. Keep a separate test set you never look at during development. If scores on the new set are much lower, the prompt was overfitted to examples and is not ready for wider use.

Conclusion

Evaluation is not a luxury for large teams. With 40 cases, a four-dimension rubric, and simple automated checks, a small team can make evidence-based decisions and avoid silent regressions.

A four-criteria rubric

To keep scoring consistent, rate each output on four things: correct (no factual errors), complete (answers the whole request), on-format (length and structure as asked), and reasonable on cost (no padding).

Score each 1–5 and add them up. This rubric makes two reviewers reach similar conclusions and makes comparing models easier.

Running evaluations regularly

  • Keep 20–30 real tasks as a fixed test set.
  • Run the same set whenever you change a model or prompt.
  • Record the evaluation date; the model behind the API can change without notice.

Sources & References

An example three-aspect rubric: Score one to five for accuracy, completeness, and clarity. Consistent scores over time beat gut-feel ratings.

Frequently Asked Questions

Does BankAI support streaming responses?

Yes, BankAI’s endpoint supports streaming (Server-Sent Events) and non-streaming per the OpenAI standard.

Are there RPM/TPM limits?

As long as your token quota lasts, usage is not tightly capped on TPM/TPD/RPM, but safeguards keep the service stable for everyone.

How do I choose the right context window?

Match it to the combined prompt + history + answer you need; larger-context models help with long documents.

Sources & References

About the author

Darus S Alfatah — Founder & Writer

Darus S Alfatah is the founder of BankAI. He builds and operates the LLM API gateway that serves Indonesian developers, and writes about LLM integration, gateway architecture, and using AI practically and cost-effectively. Every guide is tested in a real environment — including code, configuration, and token calculations — before it is published, and updated when provider documentation or pricing changes.

Editorial policy

Related articles

🚀 Ready to Use the All-in-One API Key?

Sign up, top up with QRIS, and start building with ChatGPT, Claude, DeepSeek and every other model through one key.

🔑 SIGN UP NOW · Pricing