9.5 Synthetic Instructions & Self-Instruct

Synthetic instructions expand coverage when expert-written examples are scarce. They do not remove the need for humans; they move human effort toward specifications, gold sets, difficult slices, and audit. A useful pipeline treats every generated sample as untrusted data with reproducible lineage.

Three Generation Patterns

Self-Instruct bootstraps from a small seed set by generating instructions, instances, and responses, then filtering near duplicates [1]. Evol-Instruct mutates existing instructions in depth or breadth to shape difficulty and coverage [2]. Magpie uses the pre-query state of an instruction-tuned model to sample plausible user queries from its learned conversational distribution [3].

These patterns solve different problems. Self-Instruct grows task families, Evol-Instruct controls transformations, and Magpie samples a teacher-specific prompt prior. None guarantees correctness, novelty, safety, or product relevance.

The Template Is a Generation Boundary

Magpie-style synthesis is model-specific. Do not paste literal <|user|> markers into a model whose tokenizer defines another protocol. Use tokenizer.apply_chat_template for both generator and judge, and pin their templates independently.

Query generation must have a stop token boundary at the next role or end-of-turn marker. Otherwise the teacher may continue into an assistant turn and the pipeline may mislabel part of an answer as the user query. Then render the completed user message again with add_generation_prompt=True to generate the answer.

The sketch below is deliberately framework-neutral and requires adapter code for the exact inference backend. It illustrates the contract, not a production-ready loop.

def render_query_prefix(tokenizer):
    # Template-specific support is required for a truly empty user turn.
    return tokenizer.apply_chat_template(
        [{"role": "user", "content": ""}],
        tokenize=True,
        add_generation_prompt=False,
    )

def render_answer_prefix(tokenizer, query):
    return tokenizer.apply_chat_template(
        [{"role": "user", "content": query}],
        tokenize=True,
        add_generation_prompt=True,
    )

# Resolve these IDs from the pinned generator tokenizer/template.
query_stop_token_ids = resolve_next_role_or_eot_ids(generator_tokenizer)
query_ids = generate(
    render_query_prefix(generator_tokenizer),
    stop_token_ids=query_stop_token_ids,
    max_new_tokens=512,
)
query = decode_generated_span_only(generator_tokenizer, query_ids)
answer_ids = generate(
    render_answer_prefix(generator_tokenizer, query),
    stop_token_ids=[generator_tokenizer.eos_token_id],
    max_new_tokens=1024,
)

Never recover generated text by slicing a decoded string at len(prompt): tokenizer normalization and Unicode can make character offsets disagree. Retain token boundaries and decode only generated token IDs.

Generate, Verify, Judge, Audit

A robust pipeline has independent gates:

1. schema and boundary checks: roles, EOS, language, length, empty fields, tool schema;
2. deterministic verification: execute code in a sandbox, check math with a verifier, validate JSON and tool arguments;
3. deduplication and evaluation quarantine: exact, near, semantic, benchmark contamination, private prompt neighbors;
4. judge scoring: target-domain rubric with a separately pinned judge template;
5. human gold audit: random samples plus all high-risk, uncertain, and judge-disagreement cases.

A reward threshold is not universal. Calibrate accept/review/reject regions against human gold labels and report precision by language, domain, length, difficulty, and safety slice. Use multiple signals when possible; a fluent judge can approve a wrong program.

Lineage and Mixture Control

For every candidate, retain:

• teacher model revision, tokenizer and chat-template hashes;
• generator prompt or seed task, mutation chain, random seed, and decoding settings;
• judge/verifier model and rubric versions, raw scores, execution result, and rejection reason;
• source license/consent, PII/secrets/safety decisions, and stable sample/cluster IDs;
• generation timestamp, pipeline code/config/container, and parent sample IDs.

Track yield by slice at every gate. A high overall yield can hide that safety, minority-language, or difficult examples are being discarded. Mix human-authored, verified synthetic, adversarial, and retention data with explicit sampling weights. Do not let the teacher’s default verbosity or favorite topics define the student distribution.

Keep related prompts, responses, mutations, and synthetic siblings in one semantic cluster before split assignment. Quarantine public benchmarks, private evaluations, their rubrics, semantic neighbors, and teacher-generated variants. A generated paraphrase of a benchmark is still contamination.

Preventing Recursive Degradation

Repeatedly training on model-generated samples can shrink coverage or amplify artifacts, but “model collapse” is not a deterministic path to a single point. Risk depends on how synthetic and real data are selected, mixed, and evaluated [4].

Mitigations include fresh human data, explicit rare-slice targets, source diversity, calibrated rejection sampling, semantic diversity metrics, and fixed human-authored retention evaluations. Temperature alone does not restore missing tails; it can also increase errors.

Before training, compare the synthetic mixture with a human-only baseline at the same token budget. After a small pilot, require improvements on target slices without exceeding base-retention, safety, and diversity regression limits. Preserve the prior manifest and model bundle for rollback.

Synthetic instruction data is also not a substitute for diagnosing the intervention. Use retrieval for changing attributable knowledge, continued pre-training for broad distribution adaptation, and SFT for supervised behavior when held-out evidence supports that choice.

Quizzes

References

1. Wang, Y., et al. (2022). Self-Instruct: Aligning Language Models with Self-Generated Instructions. arXiv:2212.10560.
2. Xu, C., et al. (2023). WizardLM: Empowering Large Language Models to Follow Complex Instructions. arXiv:2304.12244.
3. Xu, Z., et al. (2024). Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. arXiv:2406.08464.
4. Shumailov, I., et al. (2024). AI models collapse when trained on recursively generated data. Nature.