Tokenizer Design Notes for a Shared Origin–NextGen Vocabulary
How we chose a byte-level BPE vocabulary for code-heavy workloads and future multimodal text interfaces—fertility measurements, code round-trip tests, and the case for boring infrastructure done right.
Abstract
Origin and NextGen share a tokenizer. This is a deliberate architectural decision that enables distillation (teacher and student operate in the same token space), routing transparency (Quark can compare logit distributions), and consistent decoding semantics across the model family. These notes capture the 2023 design choices, the candidate evaluation methodology, and the fertility and quality tradeoffs that led to the production vocabulary.
The punchline is unglamorous: byte-level BPE won on robustness and code tasks at a modest fertility penalty for some morphologically rich languages—a tradeoff we accepted for 2023 priorities.
1. Requirements
We specified five requirements before evaluating any candidate tokenizer:
- Strong code tokenization — minimal pathological splits on identifiers, keywords, operators, and indentation. Python, JavaScript, TypeScript, Rust, C++, SQL, and shell scripts are primary targets.
- Stable UTF-8 handling — all byte sequences must be representable; no undefined behavior on arbitrary Unicode inputs.
- Reasonable fertility on English prose — fertility is tokens per word. High fertility means more tokens to represent the same text, increasing sequence length and serving cost. English prose should be competitive with existing production vocabularies.
- Expandable special token space — reserved ranges for system/user/assistant roles, tool call delimiters, and future multimodal sentinels (image boundaries, audio boundaries) that NextGen will use.
- Shared across Origin and NextGen — the tokenizer is fixed before either model is trained; vocabulary changes after training are prohibitively expensive.
2. Candidates Evaluated
We evaluated four tokenizer families:
| Candidate | Description |
|---|---|
| WordPiece-style | Word-split with BPE subwords; similar to BERT-era models |
| Character-heavy BPE | Small vocabulary, many characters; very high fertility |
| Byte-level BPE (BBPE) | BPE over raw UTF-8 bytes; no OOV; all sequences representable |
| Unigram LM | Probabilistic segmentation; variable length |
Each candidate was trained at vocabulary sizes of 32K, 64K, and 100K on IdonAI's filtered pretraining corpus (see Data Quality Filtering v1).
3. Evaluation Methodology
3.1 Fertility
Fertility was measured as mean tokens per word on held-out sets:
- English prose (news + books)
- Python code (GitHub repositories, filtered)
- JavaScript code
- Scientific text
- Three non-English languages: French, Chinese, Arabic
Lower is better; lower fertility means shorter sequences, lower serving cost, and more content per context window.
3.2 Code round-trip test
We tokenized and detokenized 10K code files across five languages and measured:
- Exact round-trip rate: decoded bytes must equal input bytes
- Identifier preservation rate: named identifiers must not be split at arbitrary boundaries
WordPiece-style tokenizers failed identifier preservation at rates of 8–15% depending on identifier length—unacceptable for a code-primary model. BBPE achieved 100% exact round-trip by construction.
3.3 Downstream loss comparison
To measure real impact, we ran 500M-token continued pretraining from a fixed checkpoint with each tokenizer candidate and measured bits-per-byte on held-out code and prose after identical token budgets.
4. Results
4.1 Fertility comparison (English + code)
| Tokenizer | English prose | Python code | JavaScript | French |
|---|---|---|---|---|
| WordPiece-32K | 1.31 | 1.44 | 1.52 | 1.48 |
| BBPE-32K | 1.29 | 1.22 | 1.28 | 1.59 |
| BBPE-64K | 1.23 | 1.18 | 1.23 | 1.51 |
| BBPE-100K | 1.19 | 1.16 | 1.21 | 1.46 |
| Unigram-64K | 1.26 | 1.31 | 1.37 | 1.41 |
BBPE at 64K and 100K achieved the best fertility on code across all languages tested. English prose was competitive. French fertility was notably higher with BBPE than Unigram—a documented cost of byte-level tokenization for languages with accented characters and longer morphological inflections.
4.2 Round-trip and identifier tests
| Tokenizer | Exact round-trip | Identifier preservation |
|---|---|---|
| WordPiece-32K | 99.1% | 84.3% |
| BBPE (any size) | 100% | 100% |
| Unigram-64K | 99.4% | 91.2% |
BBPE's construction guarantees exact round-trip; all other approaches had failure cases.
4.3 Downstream loss
At matched 500M token training budgets, BBPE-100K produced lower bits-per-byte than all alternatives on code:
| Tokenizer | BPB (code) | BPB (English prose) |
|---|---|---|
| WordPiece-32K | 1.294 | 1.041 |
| BBPE-64K | 1.261 | 1.038 |
| BBPE-100K | 1.248 | 1.031 |
| Unigram-64K | 1.278 | 1.036 |
BBPE-100K won on both metrics by a margin sufficient to justify the vocabulary size increase.
5. Vocabulary Size Decision
We evaluated 32K, 64K, and 100K. Larger vocabularies reduce fertility (shorter sequences) but increase embedding table memory and slow the final linear projection at decode time.
| Vocabulary size | Fertility benefit | Embedding overhead | Decode speed impact |
|---|---|---|---|
| 32K | Baseline | Baseline | Baseline |
| 64K | ~5% shorter sequences | +2× embedding table | ~3% slower decode |
| 100K | ~8% shorter sequences | +3.1× embedding table | ~5% slower decode |
At Origin's serving load, the decode speed cost of 100K is acceptable—we verified this in microbenchmarks. The fertility improvement translates directly to shorter sequences, lower KV-cache pressure, and more effective context window use. We selected 100K vocabulary as the production size.
6. Special Token Design
We reserved structured ranges in the vocabulary for operational tokens:
| Category | Tokens reserved | Notes |
|---|---|---|
| Conversation roles | <|system|>, <|user|>, <|assistant|> | Multi-turn structure |
| Tool call delimiters | <|tool_call|>, <|tool_result|> | Structured tool invocation |
| Multimodal sentinels | <|image_start|>, <|image_end|>, <|audio_start|>, <|audio_end|> | Activated in NextGen |
| Padding / EOS | <|endoftext|>, <|pad|> | Standard |
| Future reserved | 256 tokens | Expansion buffer |
The multimodal sentinel tokens are inert in Origin (text-only) but present in the vocabulary, ensuring that Origin and NextGen share identical token IDs for all special tokens. This is required for distillation—teacher and student must agree on token semantics.
7. Non-English Fertility Cost
The accepted tradeoff: BBPE-100K has higher fertility on morphologically rich languages (Arabic, Turkish, Finnish) than Unigram-based alternatives. A 100K vocabulary covers common Arabic sequences reasonably well but tokenizes rare forms into many byte tokens. This imposes a real cost on non-English content quality relative to English, and is a known limitation of the 2023 vocabulary.
Multilingual fertility improvements are a target for future vocabulary revisions—though vocabulary changes require full model retraining and are expensive to make post-hoc.
8. Shared Tokenizer Benefits
The decision to share a single vocabulary across Origin and NextGen pays dividends in three areas:
- Distillation: Teacher (NextGen) traces are directly usable as SFT data for Origin without re-tokenization or token-space mapping. The distillation study relies on this property.
- Routing: Quark's router can compare logit distributions across models when they share a vocabulary—enables uncertainty-based escalation metrics.
- Infrastructure: A single tokenization service handles all IdonAI model traffic; no per-model tokenization code paths.
9. Limitations
The 100K BBPE vocabulary has higher fertility costs on non-English content than alternatives. It was designed and evaluated primarily on English and major programming languages; performance on low-resource languages and non-Latin scripts is weaker. The vocabulary is fixed—changing it requires retraining all models from scratch.
10. Conclusion
A shared vocabulary is infrastructure, not research. But infrastructure decisions made early become permanent constraints. The choice of BBPE-100K—made in early 2023 before either Origin or NextGen was fully trained—has been validated by every subsequent training run. Code tokenization quality, exact round-trip guarantees, and distillation compatibility all depend on this decision.
Boring infrastructure done right compounds over time. Boring infrastructure done wrong is a multi-year tax.