10.1 Human Feedback Loop
Human feedback is a measurement system before it is a training dataset. Product feedback records what users were shown, what they chose to signal, and under which interface and policy. Without exposure and sampling context, a thumbs-up is not an unbiased label of model quality.
Offline and Online Feedback
Offline programs collect controlled demonstrations, rankings, critiques, ties, and abstentions under a rubric. They support balanced sampling and repeat annotation. Online programs observe deployed interactions and can find real failures, but contain exposure, position, engagement, survivorship, and selection bias.
Log the candidate set, displayed order, serving policy, model/artifact revision, decoding settings, user-visible latency/errors, and sampling or propensity probability. Run randomized A/B or interleaving experiments when causal product decisions require them. Do not infer that the answer receiving more clicks is intrinsically better if it was shown more often or in a favored position.
An admission score is a routing aid, not a license to train only on high-reward trajectories. Selecting only successful or high-scoring outputs narrows support and creates selection bias; retain controlled negatives, counterfactual comparisons, and exploration data.
Annotation Contract
Write a task-specific rubric with examples of correct, partially correct, unsafe, overly refusing, unverifiable, and out-of-scope answers. Let annotators mark tie when responses are equivalent and abstain when evidence or expertise is insufficient. Forcing every item into a binary preference manufactures noise.
Calibrate annotators on a human gold set, measure inter-annotator agreement by domain and difficulty, and adjudicate critical disagreements. Store rubric version, randomized position, annotator qualification, timestamps, confidence, and reason codes. Monitor systematic differences across demographic, language, accessibility, and domain slices without using sensitive attributes beyond consented policy.
Privacy is part of the data contract. Define consent, retention, access, deletion, and reuse scope; detect and redact PII, secrets, credentials, and third-party content before annotation or training. Keep raw production logs outside the training corpus until approved transformation produces a versioned manifest.
Active Learning Without Feedback Loops
Spend review budget where labels change a decision: policy uncertainty, judge disagreement, high-impact domains, novel clusters, safety-critical actions, and distribution drift. Include a probability sample of ordinary traffic so the dataset does not become only hard cases.
Log the sampling probability and use stratification or propensity-aware analysis when estimating population metrics. Active-learning data can be excellent for training while being invalid for an unweighted product-quality estimate.
For verifiable tasks, pair human preference with executable evidence: unit tests, tool-state checks, citations, or environment outcomes. A reward model should not overrule a failed deterministic verifier merely because the explanation is fluent.
The Badcase Loop Without Leakage
A production failure should first become a reproducible evaluation case, not immediately a training example:
- freeze prompt, context, tool state, expected behavior, artifact revision, and failure taxonomy;
- assign its semantic cluster and create a private evaluation item;
- create separate sibling holdout variants that remain evaluation-only;
- only then add remediated examples or other siblings to a versioned training candidate;
- rerun the original, sibling holdout, capability-retention, and safety suites.
This prevents a fix from “passing” by memorizing the exact incident. Keep an untouched private release set and rotate access, not the answers, through training systems.
Admission, Training, and Release Gates
Define separate thresholds for data admission and model release. For data, track source yield, ties/abstentions, agreement, judge disagreement, PII rejection, and slice coverage. For training, pin the manifest, split clusters, reference/baseline artifact, tokenizer/templates, objective, and seeds. Save optimizer/scheduler/scaler/RNG/sampler/data cursor with the checkpoint.
For release, use fixed decoding and paired comparisons with uncertainty. Gate target quality, base retention, safety/over-refusal, format/tool correctness, latency, cost, and error rate; critical slice failures cannot be averaged away. Shadow, canary, and ramp-up stages need automatic abort windows and a rehearsed last-known-good rollback.
Quizzes
Quiz 1: Why is a production thumbs-up not an unbiased quality label?
The user saw a selected model output under a particular position, latency, interface, and policy, and chose whether to respond. Exposure, engagement, and selection processes all affect the observed label.
Quiz 2: Why allow ties and abstentions?
Equivalent answers have no honest direction, and some comparisons exceed the annotator’s evidence or expertise. Forcing a winner converts uncertainty into systematic label noise.
Quiz 3: Why create a sibling holdout before training on a badcase?
The exact incident can be memorized. Evaluation-only siblings test whether the fix generalizes to the failure cluster rather than merely reproducing one expected answer.
Quiz 4: Why retain random traffic alongside active-learning samples?
Uncertainty sampling overrepresents hard and unusual cases. A probability sample supports unbiased population estimates and reveals whether improvements transfer to ordinary traffic.
References
- Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
- Bai, Y., et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862.
- Casper, S., et al. (2023). Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. TMLR.