17.4 Contamination Issues

Evaluation contamination occurs when training or tuning data contains test items, solutions, rubrics, close paraphrases, or information produced from them. It inflates measured performance without proving transferable capability. Exact matching is one layer of defense, not a verdict.

Contamination Lineage

Assign every protected case a stable evaluation ID, normalized hash, raw-source hash, semantic cluster ID, version, and access class. The protected record includes prompt, answer, rubric, grader/judge prompt, fixtures, unit tests, tool environment, and any public aliases.

Quarantine not only the item but also:

• answer explanations, benchmark repositories, leaderboards, and solution discussions;
• translations, format conversions, perturbations, and semantic neighbors;
• private release prompts and prior production badcases;
• synthetic outputs created with the item, answer, teacher prompt, or rubric in context.

Track parent-child lineage through teacher model revision, prompts, seeds, decoding, judge/verifier versions, and rejection reasons. A generated “new” question can still be a benchmark derivative.

Detect at Multiple Representations

Use a cascade on raw, rendered, and tokenized forms:

1. exact hashes after conservative normalization;
2. document and substring matching;
3. n-gram, MinHash/LSH, or suffix/prefix overlap;
4. code AST, identifier-normalized, or test-fixture similarity;
5. embedding retrieval and cluster review for semantic neighbors.

An overlap score has no universal cutoff. Calibrate the threshold on known duplicate, paraphrase, and legitimate-domain-neighbor pairs for each language, tokenizer, document length, and data type. Report precision/recall and send the uncertain band to review. A fixed number such as 13 matching tokens or 0.8 similarity is not a general boundary.

Run the scan on raw documents and again on the final rendered and tokenized corpus after filtering, chat templating, packing, and synthetic augmentation. Post-processing can introduce or expose matches that raw-document scans miss.

Decontamination Pipeline

The protected registry is versioned before data collection. Every pretraining, continued-pretraining, SFT, preference, reward-model, and synthetic manifest records the registry version it checked.

For each candidate sample:

1. retain source provenance and stable sample ID;
2. query exact and approximate indices;
3. attach matched evaluation IDs, distances, and evidence;
4. remove, quarantine, or escalate according to a declared rule;
5. record the decision and all descendants so deletion propagates;
6. recompute mixture weights after removal.

Split semantic clusters before train/dev/test assignment. Never tune thresholds by looking at desired benchmark scores. Keep contamination reviewers and training curators from silently overriding protected cases without an audit trail.

Dynamic and Private Evaluation

Fresh private cases, timestamped tasks, perturbations, and dynamic user prompts reduce some direct leakage. They do not guarantee contamination-free evaluation. Prompts may later be logged or published; teachers can transfer solutions; semantic templates can be known; judges can have seen rubrics; repeated online evaluation can become training feedback.

Treat dynamic benchmarks and arenas as useful changing samples, not a complete safeguard. Record item creation and first-exposure dates, system/tool access, judge version, and whether public logs are reusable. Combine stable anchors for trend measurement with protected fresh cases and product-specific private suites.

Suspicious behavior includes exact continuation of hidden text, brittle answer-label dependence, abnormal performance on known/public versus fresh variants, and performance drops under meaning-preserving transformations. These are diagnostic signals, not proof by themselves.

Reporting and Release Gate

A benchmark report states the data cutoff claim, protected-registry version, scanned manifests, representation levels, thresholds and calibration set, exclusions, residual uncertainty, and whether training providers exposed sufficient lineage. “No match found” means no match under the tested detector.

Block release claims when protected evaluation IDs or semantic neighbors entered training, when teacher/judge lineage is unavailable for material synthetic data, or when the candidate gains only on exposed cases. Rebuild from a clean manifest where feasible and evaluate on untouched siblings.

Quizzes

References

1. Golchin, S., & Surdeanu, M. (2023). Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models. arXiv:2311.06233.
2. Sainz, O., et al. (2023). NLP Evaluation in Trouble: On the Need to Measure LLM Data Contamination for Each Benchmark. arXiv:2310.18018.
3. Lee, K., et al. (2021). Deduplicating Training Data Makes Language Models Better. arXiv:2107.06499.