9.1 Supervised Fine-Tuning (SFT) Fundamentals

A base model learns broad statistical structure by predicting tokens. Supervised Fine-Tuning (SFT) changes its behavior with curated demonstrations: answer this kind of request in this format, follow this tool schema, or refuse this unsafe request. SFT can also teach task knowledge when examples contain it, but it is not a reliable replacement for retrieval when facts change or require attribution.

The practical objective is simple. The data contract is not.

Objective and Loss Mask

For messages serialized into tokens $z_1,\dots,z_T$, let $m_t=1$ only when token $z_t$ belongs to an assistant span that should be learned. Completion-only SFT minimizes

Labels for the system prompt, user turns, padding, and any excluded tool spans are set to -100. This prevents those positions from contributing directly to cross-entropy. It does not remove prompt tokens from the forward pass or their role as context: assistant predictions still attend to prompt activations, so prompt masking alone does not save activation memory in proportion to prompt length. Sequence length, checkpointing, attention implementation, and packing policy determine activation memory.

Full-sequence loss is sometimes intentional—for example, when adapting a base model to a structured transcript distribution—but it is a different objective. Record which roles receive loss instead of calling one policy universally correct.

The Tokenizer and Template Are Part of the Model

Do not hand-write <|user|> strings copied from another model family. Use the checkpoint’s tokenizer-native chat_template and freeze its exact bytes or hash with the training artifact. The same template, BOS/EOS behavior, and generation configuration must be used during training, evaluation, and serving.

Before a run, inspect several encoded examples and verify:

• add_generation_prompt=False for complete training conversations and the serving-time value for inference;
• BOS is not duplicated and every learned completion has the intended EOS or end-of-turn token;
• every assistant turn—not only the last one—has the intended loss mask;
• system, user, padding, and excluded tool-result tokens are -100;
• truncation never silently removes the answer while leaving only prompt tokens;
• the rendered serving prefix is token-identical to the training prefix.

Some tokenizers expose an assistant mask only when their Jinja template contains generation blocks. If the tokenizer cannot return a trustworthy assistant mask, derive spans from tokenizer offsets or a template-aware collator and test the result. Searching token IDs for a delimiter is unsafe when the delimiter can appear in content.

Adding new special tokens is a migration, not a formatting shortcut. It requires resizing and initializing embeddings, deciding whether lm_head is tied, retraining those rows, and recording checkpoint compatibility.

A Mask-Inspection Collator

The following is a deliberately compact educational example. It assumes a recent Transformers tokenizer whose chat template supports return_assistant_tokens_mask. API details vary by tokenizer and library version; test this path against the exact pinned artifact before training.

import torch

IGNORE_INDEX = -100

def encode_conversation(tokenizer, messages, max_length=4096):
    encoded = tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=False,
        return_dict=True,
        return_assistant_tokens_mask=True,
        truncation=True,
        max_length=max_length,
    )
    input_ids = torch.tensor(encoded["input_ids"], dtype=torch.long)
    assistant_mask = torch.tensor(
        encoded["assistant_masks"], dtype=torch.bool
    )
    attention_mask = torch.ones_like(input_ids)

    labels = input_ids.clone()
    labels[~assistant_mask] = IGNORE_INDEX

    if labels.ne(IGNORE_INDEX).sum() == 0:
        raise ValueError("truncation removed every supervised token")
    if tokenizer.eos_token_id is not None:
        learned = input_ids[assistant_mask]
        if learned[-1].item() != tokenizer.eos_token_id:
            raise ValueError("assistant span does not end with the expected EOS")

    return {
        "input_ids": input_ids,
        "attention_mask": attention_mask,
        "labels": labels,
    }

def pad_batch(tokenizer, examples):
    if tokenizer.pad_token_id is None:
        raise ValueError("define a deliberate padding policy")
    width = max(x["input_ids"].numel() for x in examples)
    batch = {}
    for key, pad_value in (
        ("input_ids", tokenizer.pad_token_id),
        ("attention_mask", 0),
        ("labels", IGNORE_INDEX),
    ):
        rows = []
        for item in examples:
            pad = width - item[key].numel()
            rows.append(torch.nn.functional.pad(item[key], (0, pad), value=pad_value))
        batch[key] = torch.stack(rows)
    return batch

Visualize tokenizer.convert_ids_to_tokens(input_ids) beside the boolean mask for real multi-turn, tool-use, long, empty, and multilingual examples. This is more valuable than checking only tensor shapes.

Packing and Truncation

Packing short conversations into one sequence improves utilization, but ordinary causal attention lets later samples attend to earlier samples. Choose one of these contracts:

1. concatenate documents with explicit EOS and accept cross-sample context as part of the training distribution;
2. use block-diagonal or sequence-ID-aware attention so samples are isolated;
3. disable packing when the implementation cannot prove either behavior.

Never rely on EOS alone to block attention—it is a learned token, not an attention barrier. Log packing efficiency, supervised-token fraction, prompt/response length tails, examples dropped by truncation, and samples with zero supervised tokens.

Run Contract and Evaluation

Pin the base checkpoint, tokenizer, chat template, dataset manifest and split hashes, code/container, optimizer, precision, sequence length, packing policy, and seeds. Estimate weights, gradients, optimizer states, adapters, activations, temporary buffers, checkpoints, and evaluation separately. Save model/adapter, optimizer, scheduler, scaler, RNG, sampler, and data cursor so a resume does not replay or skip examples.

Start with a small pilot and monitor token-normalized train and validation loss, gradient norms, nonfinite counts, throughput, data wait, prompt/response length, and loss by domain. Abort or investigate when masks are empty, EOS coverage changes, validation loss regresses beyond a declared band, nonfinite events persist, or a safety-critical slice fails.

Immediate offline evaluation should include:

• held-out instruction following and exact format/tool-schema validity;
• base-capability retention and domain slices;
• safety/refusal over-refusal pairs;
• long-context and multi-turn cases matching the truncation policy;
• fixed decoding settings and paired comparisons against the frozen baseline.

Promote only the immutable artifact bundle that passed the gate. Keep the previous bundle as last-known-good and test rollback before serving traffic.

Quizzes

References

1. Zhou, C., et al. (2023). LIMA: Less Is More for Alignment. arXiv:2305.11206.
2. Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155.
3. Hugging Face. Chat templates. Transformers documentation.