Benchmark scores predict benchmark performance. They are a weak signal for production quality on your specific task. Here's how to build an evaluation harness that actually tells you something useful.
The standard model selection workflow, which involves reading the leaderboard, picking a top performer, running some informal tests, and shipping, produces confident decisions that are frequently wrong. Benchmark performance measures how a model responds to standardized questions under standardized conditions. Your production task is neither standardized nor identical to what was benchmarked. The delta between those two things is where evaluation should live.
This isn't a criticism of benchmarks as a concept. MMLU, HumanEval, MATH, and SWE-bench measure legitimate capabilities on specific tasks. The problem is that they measure those capabilities on the benchmark task distribution, not on your task distribution. A model that achieves state-of-the-art on MMLU may behave very differently from a slightly lower-scoring model when the task involves your specific domain vocabulary, your prompt structure, your output format requirements, and your edge case distribution.
Organizations deploying LLMs in production, whether for document extraction, classification, agent reasoning, or content generation, need evaluation infrastructure that tells them how models perform on their specific task, not on a standardized proxy for it.
The Benchmark Problem in Specific Terms
Popular benchmarks measure things that are measurable under controlled conditions:
- MMLU: multiple-choice question accuracy across 57 subjects, measuring broad factual recall and reasoning
- HumanEval and SWE-bench: code completion and software engineering task performance
- MATH and GSM8K: mathematical reasoning on structured problem sets
- HellaSwag: commonsense natural language inference
- GPQA: graduate-level science question answering
These are legitimate measures of capability on those specific tasks. They do not measure:
Instruction-following reliability on your prompt structure. A model that follows instructions reliably in the benchmark evaluation context (which uses a specific prompting convention) may respond differently to your system prompt structure, your output format requirements, or your few-shot examples. Instruction-following capability is general; its expression depends on how the instruction is structured, and that structure varies significantly across production deployments.
Output format consistency. Models that perform well on tasks where the output format is flexible can be inconsistent when required to produce output in a specific format: JSON with a particular schema, XML with defined tags, structured tables with specific column ordering. Consistency in format matters more in production automation pipelines than in human-reviewed tasks, and it's not measured by any standard benchmark.
Domain-specific performance. A model scoring highly on general knowledge benchmarks may perform significantly worse on documents, terminology, and entity types specific to your domain. Legal contracts, industrial control system documentation, financial filings, and specialized scientific literature all have domain-specific characteristics that affect model performance in ways that domain-general benchmarks don't capture.
Failure mode distribution. Benchmarks report accuracy: how often the model is correct. They don't characterize what happens when the model is wrong. For production systems, the nature of failure matters: a model that fails loudly (refuses, hedges, produces clearly incorrect output that's easy to detect) is significantly easier to manage than a model that fails silently (produces plausibly correct output that is subtly wrong). Benchmark scores don't distinguish between these failure modes.
Building a Task-Specific Evaluation Harness
A minimum viable evaluation harness for a production LLM task requires three components: a representative test set, a scoring function, and a baseline.
Constructing the Test Set
The test set should include examples from three categories:
Typical inputs. Representative samples from the distribution you expect in production. If you're building a document extraction system, these are typical documents from the domain. If you're building a classification system, these are typical inputs across all classes. Aim for at least 30 to 50 examples per major input category, drawn from real production data where possible rather than synthetically generated.
Known edge cases. Inputs you already know cause problems: documents with unusual formatting, edge-case phrasings that could be misclassified, inputs that fall near class boundaries, inputs with missing or ambiguous required fields. These should be drawn from domain knowledge and from any prior system you're replacing. If you have access to examples that caused problems in a prior iteration, include them.
Adversarial examples. Inputs designed to probe failure modes: malformed inputs, inputs that look like valid requests but are subtly ambiguous, extremely long inputs that approach context limits, inputs with conflicting requirements, inputs in edge-case character sets or languages. These don't need to succeed. They need to fail gracefully (escalation or clear refusal rather than confident incorrect output).
The test set should be static between evaluation runs so you're measuring model performance on the same distribution. When you add new examples (because a failure mode has been discovered in production), version the test set so you can compare results across versions and understand what the new examples changed.
Choosing the Scoring Function
The scoring function depends on what you actually care about in production.
Exact match is appropriate for tasks with unambiguous correct outputs: entity extraction with a defined schema, code generation where the output can be executed and tested, classification with discrete labels. Exact match is easy to compute and interpret, but it's strict. A response that is substantively correct but formatted slightly differently registers as wrong.
F1 at token or entity level is appropriate for information extraction tasks where partial credit matters. A model that correctly extracts 8 of 10 required fields is more useful than one that extracts 5, even though neither achieves exact match. Reporting both precision and recall separately (not just F1) gives you actionable information: low precision means the model is inventing fields, low recall means it's missing them.
LLM-as-judge is appropriate for tasks where the output is free-form text and quality is subjective: summarization, generation, explanation, multi-step reasoning. A separate model evaluates outputs against a rubric. This approach scales to large test sets without human labeling cost, but introduces its own reliability considerations. The judge model has biases and failure modes of its own. Measuring judge-model agreement with human labels on a sample of your test set calibrates how much to trust the automated scores.
Human evaluation remains the gold standard for tasks where output quality is genuinely subjective or where the cost of an error is high. Human evaluation is expensive and slow, but it doesn't inherit the judge model's failure modes. For high-stakes applications, a regular human evaluation sample run alongside automated evaluation gives you the calibration signal you need to know whether the automated scores are drifting from what humans would actually judge.
Establishing the Baseline
The baseline is consistently underestimated during evaluation design. Before comparing models, establish what a naive approach produces: a regex pattern for extraction tasks, a template for generation tasks, a logistic regression classifier trained on labeled examples for classification tasks, or even a rule-based decision tree for structured decision tasks.
If the LLM doesn't meaningfully outperform the simpler solution on your specific task, the overhead of the LLM (higher latency, higher cost, non-determinism, dependency on external APIs, prompt engineering maintenance burden) may not be justified. The baseline comparison calibrates the evaluation: if a regex achieves 85% exact match on your test set and your LLM achieves 87%, the narrow advantage warrants scrutiny before committing to the more complex and expensive solution.
Eval sets go stale. As your task evolves and your inputs shift, the examples you collected six months ago may no longer represent what's hitting production. Build a process for sampling production inputs, labeling them (automatically or manually), and adding them to the test set continuously. An eval set that stops reflecting production distribution is no longer measuring what matters.
Latency, Cost, and Reliability as First-Class Metrics
Evaluation that ignores operational characteristics is incomplete for production systems. Three metrics matter alongside quality.
Latency distribution. Measure median (p50), 95th percentile (p95), and 99th percentile (p99) latency separately. For user-facing applications, p95 and p99 determine perceived performance, not median. A model with 400ms median latency and 4000ms p99 latency will generate user complaints even though the median looks acceptable. Load test with realistic concurrency rather than sequential single-threaded calls. Latency at p99 under production concurrency is what users experience during peak periods.
Cost per task. At prototype scale, API costs are negligible. At production scale with high request volume, they're a significant line item. A model that's 20% better on quality metrics but 3x more expensive per call may not be the right choice for a high-volume pipeline. Measure input and output tokens per task separately (they're billed differently by most providers) and calculate cost per task at your expected production volume. The cost per task calculation at scale often changes which model is the right choice.
API reliability under load. Rate limits, error rates, and timeout behavior under concurrency are not visible from benchmark results and vary significantly across providers. Before committing a provider for a production use case, characterize their API's behavior at your expected volume: what is the effective throughput before rate limiting begins, what error codes appear under rate limiting, what retry behavior is required, and how does latency degrade as you approach the rate limit. These characteristics affect system architecture decisions in ways that quality benchmarks don't.
Regression Testing Across Model Updates
Models get updated. Providers ship new versions, deprecate old ones, and adjust behavior in ways they may not communicate clearly. Each update is a potential regression on your specific task, independent of whether it improves overall benchmark performance. Provider model updates have been observed to degrade performance on specific task types while improving aggregate benchmark scores.
The test suite built for initial model selection doubles as a regression detection system: run it against every new model version before switching traffic, and alert when key metrics change beyond a defined threshold. For quality metrics, a 2 to 5% change is usually significant enough to investigate before switching. For latency, a sustained 20% increase at the same concurrency level warrants scrutiny.
The practical process for model updates:
Provider announces a new version or deprecation. Run the full test set against the new version. Produce a metric comparison table: quality metrics by task type, latency distribution (p50/p95/p99), cost per task. Identify any regressions.
Investigate regressions. Reproduce the failing test cases with both the old and new model. Determine whether the regression is systematic (a specific task type consistently degrades) or sample noise (a few examples regress while the overall distribution is similar). Systematic regressions on core task types require investigation before deploying. Sample noise in edge cases may be acceptable if other metrics improve.
Shadow mode before traffic switch. Send all production requests to both the old and new model versions and compare outputs before switching traffic. Shadow mode catches regressions that weren't represented in the test set. Run shadow mode for a period long enough to cover the full input distribution variation you see in production.
Most teams learn the value of this process by experiencing a silent model update regression that surfaced through user complaints rather than through monitoring. The second time, they have the regression suite.
Connecting Evaluation to System Design
Evaluation infrastructure provides the most value when it influences system design decisions, not just model selection.
Identifying tasks that shouldn't use LLMs. A task where the LLM achieves 88% accuracy and a regex achieves 85% is a poor candidate for LLM deployment. Without evaluation infrastructure, the LLM gets deployed by default because it's what the team built first. With evaluation infrastructure, the comparison is explicit.
Informing prompt design. Running the same task with multiple prompt structures against the same test set shows which prompt structures perform best on your specific task distribution. This is more informative than A/B testing on production traffic because you control the input distribution. Prompt engineering decisions made with eval data behind them are more defensible than those made from intuition.
Setting escalation thresholds. For agent systems that should escalate when confidence is low, the evaluation process characterizes the relationship between model confidence signals and actual accuracy. If the model's confidence signals correlate poorly with accuracy on your test set, you need a different escalation mechanism.
Making the build-vs-fine-tune decision. A custom fine-tuned smaller model may outperform a larger general-purpose model on your specific task while costing significantly less per call. The evaluation harness is the tool that makes this comparison with actual data. Without it, the decision is made on intuition or provider recommendations.
Building the Infrastructure
The investment in evaluation infrastructure pays for itself when it prevents a bad model selection decision, catches a regression before it reaches production users, or provides the data needed to justify a platform change. For teams shipping production AI applications as a core business capability, that return typically arrives within the first three months of the evaluation system being operational.
The infrastructure itself is not complex: a versioned test set stored in version control, a scoring script that runs evaluations and outputs structured results, and a process for running the evaluation before deployments and model switches. The complexity comes from maintaining the test set as the task evolves, which requires a continuous process for sampling and labeling production data. That process is operational work, not engineering work, and it's the part most teams underinvest in relative to building the evaluation code itself.
Teams that build AI-powered applications as part of their product, whether that's a custom AI agent, a document processing pipeline, or an LLM-integrated workflow, need this infrastructure to make good decisions about model selection, prompt changes, and model updates over the life of the system. Treating evaluation as a one-time exercise during initial model selection misses most of the value it provides.