LLM evaluation for product features

What you’re actually evaluating

Public benchmarks tell you about models in general. They don’t tell you whether your support assistant gives correct refund answers or whether your extraction step gets dates right on your documents. Product LLM evals measure your feature, on your data, against your definition of correct.

Start by writing down what good looks like as checkable criteria. “Helpful and accurate” isn’t checkable. “Cites at least one source; states the refund window correctly for the customer’s plan; doesn’t promise a refund; under 120 words” is.

Step 1: Build a golden set

A golden set is a collection of inputs with the expected output, or the properties a correct output must have. It’s the most valuable artifact in any AI project, and it’s mostly manual work.

1. Source real inputs: production logs, support tickets, search queries, documents users actually upload. Synthetic examples are useful for coverage, but real ones carry the messiness that breaks things.
2. Cover the distribution, then the edges. Common cases in proportion, plus deliberate edge cases: ambiguous questions, out-of-scope requests, adversarial inputs, empty retrieval, very long inputs, other languages.
3. Label with a domain expert. For each case, record the reference answer or the required properties. Disagreements between labelers are useful – they reveal where your spec is vague.
4. Start at 50–100 cases and grow to a few hundred. A small, carefully labeled set beats a large, noisy one.
5. Version it alongside your code, and tag cases by category so you can see which slice regressed.
6. Keep a held-out set you don’t tune prompts against, so you can detect overfitting to the main set.

Step 2: Choose graders

Use the cheapest grader that’s reliable for each criterion. In order of preference:

• Code-based checks. Schema validity, exact match on extracted fields, numeric tolerance, required or banned strings, length, valid citation IDs, tool called with the right arguments, tests passing for generated code. Fast, free and deterministic. Use them wherever possible.
• Reference comparison. Compare against the labeled answer with exact match, field-level scoring, or semantic similarity for free text. Similarity scores are blunt; use them as a signal, not a gate.
• LLM-as-judge. A model grades the output against a rubric – faithfulness to sources, tone, whether a specific fact is stated correctly. Necessary for qualities code can’t check, and the most likely to mislead you.
• Human review. The ground truth that everything else is calibrated against. Too slow for every change; essential for setting up and spot-checking the automated graders.

LLM-as-judge pitfalls

Model judges are useful and biased in predictable ways. The known problems and what I do about them:

• Vague rubrics produce noisy scores. Ask narrow, binary questions – “Does the answer state the 30-day window? yes/no” – instead of “rate quality 1–10”. Several binary checks beat one overall score.
• Position bias. In pairwise comparisons, judges favor one position. Run both orders and only count consistent wins.
• Verbosity bias. Judges tend to prefer longer answers. Put length expectations in the rubric, and check length in code.
• Self-preference. A model may rate outputs from its own family more favorably. Where it matters, judge with a different model than the one under test.
• Judging without the evidence. For faithfulness, give the judge the retrieved sources and ask whether each claim is supported – not whether the answer “seems right”.
• Uncalibrated judges. Before trusting a judge, have a human label 50–100 outputs and measure agreement. If the judge disagrees with humans often, fix the rubric before using the scores.
• Judge drift. If the judge model changes, your scores change. Pin the judge model and re-calibrate when you update it.

Step 3: Evaluate the parts, not just the whole

End-to-end scores tell you something broke; component scores tell you what. For a RAG feature, measure retrieval (recall@k against the labeled source documents) separately from generation (correctness and faithfulness given the retrieved context). For an agent, measure tool selection and argument accuracy per step, as well as task success, steps per task and cost per task. My RAG architecture guide covers retrieval metrics in more detail.

Step 4: Gate regressions in CI

1. Run the suite on every change that can affect output: prompts, tool descriptions, model or parameter changes, retrieval configuration, chunking, and the guardrail code around the model.
2. Split the suite. A fast smoke set (20–50 critical cases, mostly code-graded) runs on every pull request. The full set, including LLM judges, runs nightly and before releases.
3. Set thresholds per category, not just overall. A 2-point drop overall can hide a 20-point drop on billing questions.
4. Account for nondeterminism. Run flaky cases several times and use pass rates; set low temperature where the feature allows it; don’t fail a build on a single noisy case.
5. Show diffs. Report which cases flipped from pass to fail, with outputs side by side. That’s what reviewers actually read.
6. Track cost and latency in the same report, so a quality win that doubles cost is a visible trade-off rather than a surprise.

Step 5: Close the loop with production

Offline evals only cover what you thought of. In production, log inputs, outputs, retrieved context and user feedback (thumbs, edits, escalations, retries). Sample conversations weekly, and run your automated graders on a sample of live traffic to catch drift.

Every real failure becomes a new golden-set case. Over time the suite becomes a precise record of everything your feature has got wrong, which is exactly what you need when deciding whether a new model is safe to adopt. Pair this with the runtime checks in LLM guardrails: evals tell you how often things go wrong; guardrails limit the damage when they do.

LLM evals checklist

• Written, checkable success criteria per feature.
• Golden set of real inputs with expert labels, tagged by category, versioned in the repo.
• Held-out set for overfitting checks.
• Code graders first; LLM judges only for what code can’t check, calibrated against human labels.
• Component metrics for retrieval and tool use, not only end-to-end.
• Smoke set gating pull requests; full suite nightly and pre-release.
• Per-category thresholds, pass/fail diffs, cost and latency in the report.
• Production sampling feeding new cases back into the set.

You don’t need a platform to start – a script, a JSONL file and a CI job will do. Eval tooling helps once you have many features and want shared dashboards. Setting this up is the first thing I do on any AI integration project, because it’s what makes every later decision measurable.

Related

• AI integration services for products already in production
• LLM guardrails for real product features
• RAG architecture for production systems
• Agentic workflows: patterns that hold up in production
• RAG vs fine-tuning: how to choose
• AI consulting services for software companies