8.3 Over-training vs Optimal-training

“Optimal” depends on the objective. A compute-optimal run seeks the best validation loss for a fixed training budget. A deployment-oriented run may spend more training tokens on a smaller model because serving it repeatedly is cheaper. A post-training team may instead value a checkpoint that adapts reliably under SFT and preference optimization. These objectives can select different parameter and token budgets.

The Chinchilla study estimated compute-optimal relationships for its models, data, optimizer, and schedule [1]. Its often-quoted token-per-parameter value is not a universal constant. Architecture, data quality, context length, optimizer tuning, and the cost assigned to training versus inference all change the decision.

1. Why Train Beyond a Compute-Optimal Point?

Training cost is paid once per run; inference cost is paid for every served token. A smaller model trained on more data can therefore be economically attractive even if that run does not minimize training FLOPs. The correct comparison is lifecycle cost at the required quality:

where $Q_{\text{served}}$ is expected served tokens and $C_{\text{operations}}$ includes evaluation, failures, storage, and engineering overhead. Use measured throughput, utilization, latency, and pricing assumptions. An open-weight model, an internal batch model, and a latency-critical API can have different optima.

Over-training may improve some pre-training and downstream metrics under controlled settings [2]. It does not guarantee that a smaller model matches a much larger model, that every task improves, or that the checkpoint remains easy to adapt.

2. The Fine-Tuning Question Is Empirical

A 2025 study reports settings in which later, more heavily trained language-model checkpoints became harder to fine-tune [3]. This is important emerging evidence, not a settled law that additional pre-training “destroys adaptability.” The result can depend on the base model, data stream, checkpoints, downstream task, fine-tuning recipe, learning rate, and evaluation.

Possible mechanisms such as progressive sensitivity, sharper local geometry, or increased feature interference are hypotheses to test. A lower pre-training loss and worse SFT result do not by themselves prove one mechanism. Conversely, one successful SFT recipe does not prove that plasticity is preserved for every task.

The visualization illustrates a possible quality–adaptability trade-off. Its regions are not universal token-per-parameter thresholds.

3. Why a Single Fisher Trace Is Not a Stop Signal

The squared-gradient sum from one batch is sensitive to loss reduction, token count, parameterization, model scale, batch composition, label quality, and precision. Calling it an empirical-Fisher trace does not make values comparable across checkpoints. Random labels make the result even less representative of downstream adaptation.

If sensitivity metrics are investigated, use a fixed held-out corpus, identical causal labels and masks, normalized token counts, the same parameter subsets, multiple batches/seeds, and confidence intervals. Track gradient/update norms and loss curvature proxies as diagnostics—not as an automatic stop rule. Confirm any warning with actual downstream probes.

4. Checkpoint Selection Protocol

Save checkpoints frequently enough to compare the adaptation frontier. Before the run, freeze the following protocol:

The SFT probe should be cheap enough to run at selected pre-training checkpoints but realistic enough to predict the intended post-training stage. Include at least one domain task, one format/tool task if relevant, general-capability retention, and safety. Tune a small learning-rate range for each checkpoint; using one unsuitable LR can falsely label a checkpoint “brittle.”

Do not use the probe’s training examples in the final private release evaluation. Split related conversations, documents, entities, and synthetic siblings before assignment and keep the release set quarantined.

5. Decision and Abort Gates

Select a checkpoint on a Pareto surface rather than by minimum pre-training loss alone:

• reject any checkpoint that fails a critical safety, privacy, or capability-retention slice;
• require domain and standardized SFT gains to exceed the uncertainty band, not merely the point estimate;
• compare the additional training cost with the measured serving savings at plausible traffic volumes;
• stop a run for nonfinite state, reproducible corruption, or predeclared regression bounds—not for an uncalibrated sensitivity number;
• keep the last-known-good checkpoint and complete optimizer/scheduler/RNG/data-cursor state so the team can resume or branch experiments.

A later checkpoint may be best for base-model inference and an earlier checkpoint best for an instruction-tuned product. Keeping both can be rational. Report the chosen artifact, downstream recipe, and lifecycle assumptions together; “trained on more tokens” is not a product-quality certificate.

6. Pilot Design Before a Large Run

Use a factorial pilot across at least several model sizes and token budgets, with multiple seeds where affordable. Keep tokenizer, data snapshot, evaluation, and quality gates fixed; tune optimizer settings per scale. Reserve the largest pilot as a forecast holdout, estimate uncertainty, and test several downstream SFT probes at intermediate checkpoints.

Log global tokens, realized mixture, training/validation loss by slice, gradient/update/activation norms, nonfinite events, throughput, MFU, data wait, checkpoint overhead, and reconstructable batch IDs. A useful run plan states in advance which checkpoint comparisons can change the final token budget and who approves that change.

The engineering lesson is not “never over-train.” It is to optimize lifecycle utility with measured adaptation and retention, and to preserve enough checkpoints and state that the choice remains reversible.

Quizzes

References

1. Hoffmann, J., et al. (2022). Training Compute-Optimal Large Language Models. arXiv:2203.15556.
2. Gadre, S. Y., et al. (2024). Language Models Scale Reliably With Over-Training and on Downstream Tasks. arXiv:2403.08540.
3. Springer, J. M., et al. (2025). Overtrained Language Models Are Harder to Fine-Tune. arXiv:2503.19206.