6.2 Tokenization Science

A tokenizer is a versioned model interface. It determines sequence length, embedding identities, document boundaries, loss positions, and how much compute each language or domain receives. Changing it after pre-training starts is an architecture migration, not a harmless preprocessing edit.

The engineering goal is not the highest average compression. It is a tokenizer that represents every supported input, behaves consistently under normalization and special tokens, allocates acceptable sequence budgets across critical slices, and remains bound to the correct checkpoint and serving stack.

1. Subword and Byte Representations

Word-level vocabularies have an open-ended unknown-token problem. Character or byte representations cover arbitrary input but often produce longer sequences. Subword tokenization balances embedding-table size against sequence length by keeping frequent strings intact and decomposing rarer strings.

• BPE repeatedly merges symbol pairs according to a learned rule. Byte-level BPE begins from 256 byte values, so arbitrary byte input has a representable fallback path [1].
• WordPiece uses a likelihood-inspired merge score and is associated with models such as BERT [2].
• SentencePiece is a tokenizer toolkit that can train Unigram or BPE models directly from raw sentences without a required language-specific pre-tokenizer [3]. Its common whitespace marker is ▁, not an underscore.

“Byte-level model” and “state-space model” describe different axes. Mamba is an architecture and is not inherently byte-level. Likewise, using bytes does not eliminate segmentation policy: patching, pooling, and sequence-length limits still determine what the model processes. Byte-level systems such as MEGABYTE use architecture-specific hierarchy to manage longer sequences [4].

2. Vocabulary and Compute Trade-offs

For hidden width $d$ and vocabulary size $V$, an untied input embedding and output projection contain approximately $2Vd$ parameters. A larger vocabulary may reduce sequence length but increases parameter, optimizer, checkpoint, and softmax costs; rare rows may receive too few updates. A smaller vocabulary reduces those matrices but can allocate far more tokens—and therefore attention/activation compute—to some languages, code styles, or identifiers.

Measure the complete distribution rather than a single English average:

• tokens per byte/character and tokens per document, including p50/p95/p99;
• fragmentation of Korean, other supported languages, code, math, URLs, identifiers, and domain terms;
• frequency and document coverage of every vocabulary row;
• byte fallback or unknown-token rate and the inputs that trigger it;
• sequence truncation rate at the intended context window;
• embedding/output parameter and optimizer-state cost under the selected precision and sharding.

A “glitch token” is an operational symptom, not a universal story about one corpus. Reserved, malformed, or extremely rare token IDs can have poorly trained embeddings or unusual behavior. Audit token frequency against the actual pre-training manifest and probe rare/special tokens before launch.

3. Normalization and Special-Token Contract

Normalization can erase distinctions that matter in code, identifiers, accents, or security-sensitive text. Record Unicode normalization, whitespace handling, control-character policy, byte fallback, and invalid-input behavior. Test encode → decode round trips at the byte or declared text-normalization level and retain offset mappings for tasks that point back to source spans.

Reserve BOS, EOS, padding, unknown/fallback, mask, role, tool, and document-boundary tokens deliberately. The same numeric ID must never mean different things across tokenizer versions. Check whether the model expects BOS, whether EOS closes every document or conversation, whether padding aliases EOS, and which tokens contribute to the loss. A chat template is part of this contract even for a base model that will later be instruction-tuned.

4. Tokenizer Acceptance Test

Build the acceptance corpus before training the tokenizer. It should be a fixed, licensed, human-audited sample from every supported language/domain plus adversarial inputs: empty strings, whitespace variants, combining characters, emoji, invalid bytes under the declared policy, long identifiers, code indentation, formulas, URLs, and every special token. Keep public benchmarks and private release prompts in evaluation quarantine; acceptance examples should test the same phenomena without copying protected evaluation items.

The following deliberately small test checks interface invariants for an already loaded tokenizer. Production acceptance also needs large slice reports and manual inspection.

from collections import Counter

def audit_tokenizer(tokenizer, samples):
    lengths = []
    token_counts = Counter()
    unk_id = tokenizer.unk_token_id

    for sample in samples:
        encoded = tokenizer(
            sample,
            add_special_tokens=False,
            return_offsets_mapping=True,
        )
        ids = encoded["input_ids"]
        offsets = encoded["offset_mapping"]
        decoded = tokenizer.decode(ids, skip_special_tokens=False)

        assert len(ids) == len(offsets)
        assert all(0 <= start <= end <= len(sample) for start, end in offsets)
        assert unk_id is None or unk_id not in ids, f"unexpected UNK: {sample!r}"
        # Compare after the tokenizer's documented normalization, not blindly.
        assert decoded == sample, (sample, decoded)

        lengths.append(len(ids))
        token_counts.update(ids)

    assert lengths and max(lengths) > 0
    return {"lengths": lengths, "token_counts": token_counts}

samples = ["한국어와 English", "def f(x):\n    return x + 1", "é é 🙂"]
# report = audit_tokenizer(tokenizer, samples)

Some tokenizers intentionally normalize text, so the exact round-trip assertion must be replaced with a comparison against the documented normalized form. Do not weaken the test silently; encode the policy explicitly. Verify offsets using the framework’s exact offset unit, which may be characters or bytes.

Set slice-specific gates before choosing a vocabulary. Compare candidate tokenizers on the same corpus and include uncertainty or multiple samples where the corpus is small. Reject a candidate for unexpected unknowns/fallback failures, special-token collision, unacceptable critical-slice truncation, missing offsets, excessive tail fertility, or vocabulary rows with implausibly low exposure.

5. Checkpoint Compatibility Contract

The model, tokenizer, and template form one immutable artifact bundle. Store tokenizer files and hash, vocabulary size, token-to-ID map hash, special-token IDs, normalization policy, chat template, BOS/EOS/padding policy, and model embedding/output shapes. Serving should fail closed when these identities do not match.

Default to preserving the original tokenizer during continued pre-training or fine-tuning. Adding tokens is justified only by measured coverage or efficiency failures. Append new IDs without reassigning existing ones, initialize their input and output rows deliberately, call resize_token_embeddings, and ensure the new rows are trainable. Hugging Face explicitly requires resizing model embeddings after vocabulary expansion [5].

Do not load an old checkpoint into a reordered vocabulary merely because the matrix shape matches. The rows would refer to different strings. For tied embeddings, verify the tie remains intact after resizing; for untied heads, initialize and checkpoint both matrices. Record merge/export behavior for adapters and quantized models, where resizing may require returning to an appropriate floating-point base artifact.

6. Training and Release Runbook

Before a large run:

1. Freeze the tokenizer-training data manifest, code/config, seed, and license/PII decisions.
2. Train multiple vocabulary/normalization candidates on comparable data.
3. Run the acceptance report by language/domain and manually audit tails and rare tokens.
4. Estimate parameter, optimizer, activation, sequence-length, checkpoint, and serving costs.
5. Run a small pre-training pilot; measure validation loss by slice, rare-token exposure, throughput, truncation, and downstream probes.
6. Freeze the chosen tokenizer hash before producing final tokenized shards.
7. Gate every model load and deployment on the tokenizer/template artifact identity.

Abort or re-tokenize before scale if a critical domain depends on unknown tokens, special IDs collide, source reconstruction fails under the declared normalization, the token budget materially violates the compute plan, or the serving artifact cannot prove compatibility. Once a large model has learned token identities, replacement is normally more expensive and risky than accepting a modestly imperfect tokenizer.

Quizzes

References

1. Radford, A., et al. (2019). Language Models are Unsupervised Multitask Learners. OpenAI report.
2. Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ACL Anthology.
3. Kudo, T., & Richardson, J. (2018). SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. ACL Anthology.
4. Yu, L., et al. (2023). MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. arXiv:2305.07185.
5. Hugging Face. Tokenizer documentation. Transformers documentation.