17.5 Production Evaluation and Release Gates

A model is ready when a specific immutable artifact passes predeclared offline and online gates—not when one benchmark score rises. The release process must make the decision reproducible, identify who can approve it, and switch back safely when reality differs from the lab.

Immutable Artifact Bundle

Evaluate and deploy one immutable artifact bundle containing hashes for:

• base model and every adapter or merged checkpoint;
• tokenizer, special-token map, chat template, and generation config;
• tool schemas, retrieval/index revision, safety policy, and prompt assets;
• code commit, resolved config, container, CUDA/framework/kernel versions;
• training/evaluation data manifests, objective, and decoding seeds.

A champion/candidate registry records bundle identity, owner, intended traffic, approvals, evaluation report, and last-known-good predecessor. Reject base/adapter/tokenizer/template incompatibility before loading. A model-name string is not an artifact identity.

Release Packet and Statistical Decision

Freeze the baseline hash, prompts, rubrics, judge/verifier artifacts, tool state, fixed seeds, and decoding settings. The packet reports raw per-example results and:

• target/domain, base-retention, safety/over-refusal, format/tool, privacy/security slices;
• response length, entropy, calibration, hallucination/citation evidence;
• latency p50, p95, p99, throughput, cost, OOM/error/tool-retry rates;
• slice sample counts and critical failures that an average must not hide.

For paired outcomes, use a paired bootstrap confidence interval or paired permutation/randomization test. Predeclare minimum effect, alpha/error tolerance, sample size or power rationale, tie handling, and slice minimums. Inspect disagreements and adjudicate a sample; statistical significance does not establish product significance.

LLM judges require human calibration on the target rubric, position randomization, and agreement/error analysis by slice. Executable or rule-based verifiers take precedence when they directly test the requirement.

Gate Decision Table

Every metric maps to an action, not merely a dashboard color.

The gate definition includes owner, data window, threshold, confidence rule, missing-data behavior, and exception approver. Missing evidence is not a zero and not a pass.

Shadow, Canary, and Ramp-up

These stages answer different questions:

1. Shadow: candidate receives mirrored inputs but cannot produce user-visible output or external side effects. Compare outputs, latency, and tool plans.
2. Canary: a small randomized eligible cohort receives the candidate. Log exposure/propensity and compare against champion concurrently.
3. Ramp-up: increase traffic in explicit steps only after each observation window passes.

Use a traffic feature flag or registry pointer, not an ad hoc redeploy. Configure automatic abort thresholds for safety, error, latency, cost, and critical task failures. Add an observation window and hysteresis so noise neither flips versions repeatedly nor delays a severe stop.

Rollback Runbook

Rollback means switching to the last-known-good complete bundle, not just replacing weights. The runbook names the incident owner and approval path and covers:

• routing switch and cache invalidation;
• incompatible adapter/base or tokenizer state;
• stateful tool side effects, partially completed workflows, and external transactions;
• queued jobs, conversation/session state, and outputs already delivered;
• audit trail, affected cohort reconstruction, notification, and follow-up evaluation.

Some side effects cannot be undone by a model rollback. Use idempotency keys, dry-run/shadow tool calls, compensating actions, and queue version tags. Conduct a periodic rollback rehearsal that measures detection and recovery time and verifies the prior bundle still loads.

Evaluation-Set Maintenance

Eval-set maintenance is a controlled data lifecycle. Assign stable evaluation IDs and cluster IDs, provenance, creation reason, owner, severity, rubric version, access class, and retirement state. Deduplicate and quarantine semantic neighbors from every training and synthetic pipeline.

When a production badcase appears, freeze it as evaluation evidence and create evaluation-only sibling holdouts before training on related examples. Add fresh slices for distribution drift, but keep a stable anchor set for longitudinal comparison. Version changes and rerun both champion and candidate when prompts, graders, tools, or schemas change.

Audit evaluation access and contamination regularly. Rotate secret cases when exposure is suspected; never silently rewrite answers to preserve a trend line.

Quizzes

References

1. Liang, P., et al. (2022). Holistic Evaluation of Language Models. arXiv:2211.09110.
2. Demšar, J. (2006). Statistical Comparisons of Classifiers over Multiple Data Sets. JMLR.
3. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge University Press.