10.4 KTO, IPO, and Preference-Loss Variants

DPO assumes paired preferences. Real feedback often contains single desirable/undesirable labels, ties, or noisy pairs. KTO, IPO, and conservative DPO variants change the statistical assumptions and loss—not the need for a pinned tokenizer, completion mask, reference policy, and held-out evaluation.

IPO: A Finite Margin Target

Identity Preference Optimization (IPO) analyzes preference optimization without relying on the same Bradley–Terry reward-model assumption as DPO [1]. A common empirical IPO form penalizes squared error between the policy-reference log-ratio margin and a finite target:

\mathcal{L}_{\text{IPO}} =\left( \left[r_\theta(x,y_w)-r_\theta(x,y_l)\right]-\frac{1}{2\beta} \right)^2,

where $r_\theta(x,y)=\log\pi_\theta(y\mid x)-\log\pi_{\text{ref}}(y\mid x)$ . The finite target discourages an ever-growing preference margin. It does not guarantee immunity to label noise, length confounds, or overfitting.

At zero margin, the gradient generally pushes toward the finite target. Interpreting a zero margin as universal update cancellation is incorrect except in specific balanced aggregates; behavior depends on the label, margin, and weighting.

KTO: Unpaired Desirable and Undesirable Feedback

Kahneman–Tversky Optimization (KTO) learns from a label attached to one completion rather than requiring a chosen/rejected pair [2]. Define

r_\theta(x,y)=\log\pi_\theta(y\mid x)-\log\pi_{\text{ref}}(y\mid x), \qquad v_\theta(x,y)=r_\theta(x,y)-z_{\text{KL}}.

Desirable and undesirable samples use opposite logistic directions and may have different weights to handle class imbalance. The important quantity $z_{\text{KL}}$ is a KL reference point, not the mean of already transformed desirability rewards.

The estimator must be documented exactly. The original construction motivates a policy-versus-reference log-ratio expectation using independent or unmatched prompt/response sampling; libraries may approximate it with a separate batch, a permuted/mismatched batch, cached policy samples, or another explicitly defined estimator, often clamped nonnegative. These choices are not algebraically interchangeable. Verify the current library definition, batch construction, distributed reduction, and gradient-detach behavior before using KTO.

An educational decomposition is:

# completion_logp uses the same prompt/completion mask, EOS, padding,
# truncation, and sum/mean convention for policy and reference.
log_ratio = policy_completion_logp - reference_completion_logp

# Estimate from a separate or explicitly unmatched batch according to
# the pinned KTO implementation. Do not replace this with rewards.mean().
kl_reference_point = estimate_nonnegative_kl_reference(unmatched_batch).detach()
value = log_ratio - kl_reference_point

loss_desirable = -torch.nn.functional.logsigmoid(beta * value)
loss_undesirable = -torch.nn.functional.logsigmoid(-beta * value)

This is a semantic sketch, not a drop-in trainer. A real implementation must state sampling and reduction across ranks.

Conservative DPO and Label Uncertainty

Conservative DPO (cDPO) treats an observed pair as potentially flipped with probability $\epsilon$ . This softens the target instead of assuming perfect labels. At $\epsilon=0.5$ , the label carries no directional preference signal and the objective favors a neutral margin; it does not imply that every per-example gradient cancels at every margin.

Use empirical repeat-label or adjudication data to choose noise assumptions. Record ties and abstentions rather than forcing them into arbitrary wins. Compare KTO when unpaired binary feedback is abundant, DPO/IPO when matched pairs are reliable, and supervised rejection or reward modeling when the product needs another behavior.

Shared Data and Run Contract

All variants inherit the DPO controls:

tokenizer-native chat template with token-inspected prompt/completion masks;
EOS, truncation, padding, variable-length, and response-length policy;
exact reference checkpoint/tokenizer/template hashes and PEFT reference semantics;
cluster split, annotator agreement, position randomization, tie/abstain, and evaluation quarantine;
token-normalized batch size, $\beta$ and class-weight sweeps, seeds, checkpoint/data cursor;
chosen/desirable and rejected/undesirable log probabilities, KL reference estimate, margins, entropy, length, held-out win/acceptability, retention, and safety slices.

Abort on nonfinite or negative estimates where the implementation promises nonnegative KL, distributed-rank disagreement, class-collapse, uncontrolled length drift, or retention/safety failure. Treat a library upgrade as an objective change until fixed-batch equivalence is demonstrated.

Quizzes

Quiz 1: What practical distinction separates KTO from DPO?

KTO can use a desirable or undesirable label for a single completion, while DPO requires two responses for the same prompt. Their sampling and loss assumptions therefore differ.

Quiz 2: Why is rewards.mean() not a valid universal KTO KL reference point?

The reference point estimates a policy-reference log-ratio expectation under a defined sampling scheme. A mean of transformed desirability rewards mixes label weighting and loss terms and generally estimates a different quantity.

Quiz 3: What does cDPO with epsilon equal to 0.5 mean?

The observed label carries no directional information, so the loss favors neutrality. It does not mean all gradients vanish for every nonzero current margin.

Quiz 4: Why must a KTO library upgrade be regression-tested?

Implementations can differ in KL batches, matching, clamping, detachment, and distributed reduction. Those details change the effective objective even when the public method name is unchanged.

References

Azar, M. G., et al. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv:2310.12036.
Ethayarajh, K., et al. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. arXiv:2402.01306.
Hugging Face. KTO Trainer. TRL documentation.