17.1 Academic Benchmarks

A benchmark is a measurement protocol: item set, prompt/template, harness, tools, decoding budget, answer extractor, grader, and statistic. A score without that protocol is not comparable evidence. Academic benchmarks are useful standardized probes, but production decisions require private tasks and operational metrics.

Frontier Benchmark Map

Use benchmark families to sample different capabilities rather than declaring one general-intelligence score.

Family	Examples	Primary capability	Common confound
Knowledge/reasoning	MMLU, GPQA	closed-form knowledge and reasoning	contamination, prompt format
Math	GSM8K, MATH	numerical and symbolic reasoning	answer extraction, tool access
Code	HumanEval, MBPP, SWE-bench	generation or repository issue resolution	hidden tests, scaffold, compute
Long context	retrieval and synthesis suites	use of distant evidence	position, truncation, retrieval setup
Agents/tools	WebArena, OS/task suites	multi-step environment success	scaffold, credentials, stochastic state
Safety	policy-specific red-team suites	compliance/refusal calibration	rubric and threat model

Always record task version/split, contamination status, system prompt, few-shot examples, tool and retrieval access, context/reasoning budget, sampling count, judge, and report date. Agentic results belong to the model-plus-scaffold system.

Multimodal Evaluation

Multimodal evaluation adds input rendering and perception dependencies. Record image resolution and tiling, video frame sampling and audio inclusion, OCR preprocessing, crop order, media compression, and whether answers require grounded coordinates or temporal evidence.

Text-only variants can reveal language priors; counterfactual media and shuffled frames test grounding. Evaluate accessibility and language slices and distinguish perception failure from reasoning or answer-format failure. Proprietary media may also impose license and retention constraints.

Metric Design

Multiple Choice and Short Answers

Accuracy is meaningful only after defining normalization, option order, scoring mode, and abstention. Compare direct generation with log-probability scoring carefully: chat templates and option-token length can alter results. Randomize option order to diagnose position bias and report calibration or selective accuracy when confidence matters.

Generated Answers

The extractor is part of the metric. Version normalization, units, aliases, numeric tolerance, citation requirements, and invalid-output handling. Inspect a stratified sample of disagreements between exact, rule-based, and LLM graders.

Code and Agents

For $n$ generated code samples with $c$ correct under hidden tests, the unbiased estimator is

\operatorname{pass@}k = 1-\frac{\binom{n-c}{k}}{\binom{n}{k}}, \qquad n\ge k.

Pass@k Calculator

Evaluate the probability of generating at least one correct solution.

Total Samples Generated (n)100

Correct Samples in Pool (c)10

Samples Evaluated (k)5

If you generate 100 samples and the model naturally gets 10 of them right, picking 5 samples at random gives you a 0.0% chance of passing the benchmark.

Report $n$ , $k$ , temperature, token/compute budget, timeout, sandbox, dependencies, test version, and retry policy. For agents, task success must include side-effect correctness and budget; a verbal claim of success is not completion.

Statistical Comparison

Use a fixed evaluation harness and paired cases for champion and candidate. For binary paired outcomes, report the difference with a paired bootstrap confidence interval or exact/approximate paired test. For continuous or judge scores, preserve pairing and cluster repeated variants. Predeclare tie handling, minimum effect, confidence interval, sample size/power rationale, and multiple-comparison policy.

A 0.5-point average gain can be noise or can hide a critical regression. Report per-domain/safety/language/tool slice with sample counts and lower bounds. Use several decoding seeds when sampling variance is part of the system and avoid treating repeated samples from one prompt as independent tasks.

Contamination and Dynamic Tests

Protect prompt, answer, rubric, solution, semantic neighbors, and teacher-generated variants in an evaluation registry. Scan the final rendered/tokenized training corpus. Public benchmarks can be useful longitudinal anchors but cannot alone establish uncontaminated generalization.

Fresh and dynamic tests reduce direct memorization risk but do not eliminate leakage through templates, judges, logs, or synthetic teachers. Use timestamped private cases and rotate exposed items while maintaining a stable anchor set. See Contamination Issues for the operational contract.

Practical Benchmark Checklist

Before publishing or using a score, verify:

exact artifact bundle and harness commit are pinned;
task version, split, prompt/template, tools, and budgets are documented;
missing/invalid outputs are preserved rather than converted to zero silently;
answer extractor, judge, tests, and environment are versioned;
paired uncertainty, sample counts, and slice minima are reported;
contamination registry and final-corpus scan are recorded;
raw outputs and adjudicated disagreements are retained;
cost, latency, error, and reproducibility are included for deployment decisions.

The final recommendation should describe a quality/cost/latency/context/openness/safety Pareto surface and call for private evaluation, not name a universal best model.

Quizzes

Quiz 1: Why is a benchmark score inseparable from its harness?

Prompts, templates, tools, budgets, extraction, and grading can change the result. The evaluated object is the model within that complete protocol.

Quiz 2: What does pass@k estimate?

Given n sampled programs and c correct ones, it estimates the probability that at least one of k samples is correct, under the benchmark’s sampling and test setup.

Quiz 3: Why use paired confidence intervals?

Champion and candidate answer the same cases. Pairing controls item difficulty and estimates the distribution of per-case differences instead of comparing two unrelated averages.

Quiz 4: Why can a dynamic benchmark still be contaminated?

Templates, rubrics, judges, or previous prompts can be exposed, and logs can become future training data. Fresh timestamps reduce one route but do not prove lineage isolation.

References

Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding. ICLR.
Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374.
Liang, P., et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110.
Jimenez, C. E., et al. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR.