10.3 Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) trains a policy from pairs of responses without fitting a separate scalar reward model or running online rollouts [1]. Its compact loss does not make the data pipeline compact: most DPO failures come from broken pair construction, mismatched templates, length effects, or an unpinned reference model.

Objective

For the same prompt $x$, let $y_w$ be preferred and $y_l$ rejected. With policy $\pi_\theta$, frozen reference $\pi_{\text{ref}}$, and temperature-like coefficient $\beta$,

The bracketed quantity is an implicit reward margin. DPO increases the policy-reference relative score of the preferred completion compared with the rejected completion. $\beta$ controls how strongly deviations from the reference translate into preference logits; its practical effect depends on data, optimizer, and log-probability convention, so run a $\beta$ sweep.

Pair and Split Contract

Each record contains one prompt, chosen response, rejected response, preference strength or tie/skip state, annotator/rubric provenance, and stable IDs. Chosen and rejected must share byte- and token-identical prompt context after rendering. Do not construct pairs by joining unrelated high- and low-score answers.

Split by prompt/conversation/entity cluster before training. Keep both completions, all annotations, paraphrases, and synthetic siblings in the same split. Randomize display position during annotation; track agreement, ties, abstentions, and length/verbosity differences. Quarantine public/private evaluations, rubrics, semantic neighbors, and teacher-generated variants.

Tokenization and Log-Probability Contract

Render both sides with the exact tokenizer artifact and apply_chat_template. Use the same system messages, role order, BOS/EOS behavior, prompt truncation, and add_generation_prompt policy. Include the intended EOS in each completion and compute log probabilities only on completion tokens.

def render_pair(tokenizer, prompt_messages, chosen, rejected):
    def render(answer):
        messages = [*prompt_messages, {"role": "assistant", "content": answer}]
        return tokenizer.apply_chat_template(
            messages,
            tokenize=True,
            add_generation_prompt=False,
        )
    return render(chosen), render(rejected)

# The collator must produce, for each side:
# input_ids, attention_mask, completion_mask, and labels (-100 outside completion).

Test the token-level boundary rather than searching decoded delimiters. Padding must be excluded by attention_mask; prompt and special tokens excluded from the completion mask; variable-length batches must normalize over the intended positions.

Sequence log probability is usually the sum of completion-token log probabilities, which matches the sequence likelihood in the derivation but can create response-length pressure. A mean changes the objective and is not a neutral fix. Log chosen/rejected lengths, compare sum and length-normalized diagnostics, control verbosity in the data, and choose the serving-relevant policy explicitly.

Truncation is pair-coupled: apply the same prompt budget to both sides, preserve the assistant boundary, and reject or flag pairs whose decisive span or EOS was cut. Never let one side keep more prompt context than the other.

Reference and PEFT Semantics

Freeze and record the exact reference checkpoint hash, tokenizer hash, template hash, precision, and log-probability implementation. A reference silently reconstructed from “the same model name” is not reproducible.

With PEFT, define whether the reference is the base with the adapter disabled, a separate frozen adapter snapshot, or cached reference log probabilities. Verify equivalence on a fixed batch. Reference-free variants optimize a different objective and must be named and evaluated as such.

Run Contract and Monitoring

Start from an SFT checkpoint that already produces valid responses. Pin the pair manifest, code/container, optimizer, batch in completion tokens, precision, max lengths, gradient accumulation, $\beta$, and seeds. Save model/adapter, optimizer, scheduler, scaler, RNG, sampler, and data cursor.

Monitor more than loss:

• chosen and rejected policy log probabilities;
• chosen and rejected reference log probabilities;
• implicit reward margin and preference accuracy;
• KL proxy to reference, entropy, response length, EOS and truncation rates;
• held-out win rate with paired uncertainty;
• domain, base-retention, safety, refusal, format/tool, latency, and cost slices.

Stop when both chosen and rejected log probabilities collapse, KL/length leaves the declared band, held-out preference stalls while retention regresses, nonfinite values persist, or a critical slice fails. Release the complete artifact bundle only after a paired baseline comparison and rollback rehearsal.

Quizzes

References

1. Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290.
2. Hugging Face. DPO Trainer. TRL documentation.