Tokenizer
Understand how text becomes an integer stream a model can consume
Suggested reading: about 17 min
Learning goal
Chapter keywords
| Keyword | Explanation | ESP32 engineering analogy |
|---|---|---|
| Vocabulary | A mapping from tokens to integer IDs. | Like a table of protocol command codes. |
| BPE | An algorithm that constructs tokens by merging frequent substrings. | Like defining frequently occurring byte fragments as shorter frame types. |
| Special token | A token carrying control meaning such as beginning, end, or role. | Like a frame header, trailer, or control word. |
| Chat template | A template that formats a message list into the string expected during model training. | Like the distinct command-frame format used by each device. |
Bridge from the previous chapter
The previous chapter established the LLM objective, historical path, and state vocabulary. This chapter starts at the input boundary and fixes exact contracts for characters, bytes, tokens, special markers, and message templates.
Historical development
The history of tokenization is a search for balance among a finite vocabulary, the need to encode any text, and the need to keep sequences short. Repeated-fragment substitution from general compression gradually became the input protocol of language models and then took on the framing of conversation roles and tool messages.
BPE begins by repeatedly replacing frequent byte pairs
Philip Gage's Byte Pair Encoding repeatedly replaced the most frequent adjacent byte pair with a new symbol, originally to compress data using a simple substitution table.
Original BPE paper ↗BPE is adapted into a subword algorithm for neural translation
Sennrich and colleagues represented rare and unseen words as subword sequences, so a fixed vocabulary no longer collapsed every unknown form into the same UNK token.
Original Subword NMT paper ↗WordPiece enters large-scale translation systems
GNMT split words into a finite set of common sub-word units, covering open-ended text with a controlled vocabulary and making subword tokenization a core part of neural language systems.
Original GNMT paper ↗SentencePiece trains directly from raw sentences
SentencePiece removes the need to split on spaces first and provides a language-independent tokenizer/detokenizer, making workflows for languages without spaces and multilingual data more consistent.
Original SentencePiece paper ↗GPT-2 adopts byte-level BPE to cover arbitrary text
GPT-2 reported using byte-level BPE, balancing byte coverage with reusable subwords and reducing the blind spots that traditional word-level vocabularies had around unusual characters.
Original GPT-2 technical report ↗Chat templates bring message structure into the tokenizer contract
Modern chat models render role/content messages into a single sequence containing control tokens. The template is stored with the tokenizer, so applications should no longer guess separators by hand.
Official Hugging Face Chat Templates documentation ↗Illustrated analogy
Ticketing and train assembly at an international station
Travelers speaking different languages and carrying emoji and code fragments arrive at the station. The ticket desk first validates their documents under one set of rules, then groups common traveling fragments into short numbered cars. The stationmaster inserts control cars for “train begins,” “passenger speaks,” and “train ends”; the model sees only the final sequence of numbers.
Where the analogy stops: The analogy can suggest that every token is a readable word, but a real token may be a word, a leading-space fragment, several UTF-8 bytes, or even a piece spanning characters. IDs have no universal meaning across tokenizers, and segmentation does not directly reveal how well the model understands a concept.
Chapter walkthrough
Trace strings back to bytes and fix normalization semantics first
Visually identical text can contain different Unicode code-point sequences, such as a precomposed character versus a base character plus a combining mark. Full-width forms, line endings, and invisible control characters also change tokens. Do not clean text separately in the application, template, and tokenizer, because training and inference will be difficult to align. Specify the normalization form, whitespace and case policy, and log boundary cases as hexadecimal code points. A byte-level tokenizer can cover arbitrary input, but that does not mean malformed UTF-8 or replacement characters should be accepted silently.
Understand the exchange between vocabulary size and sequence length
BPE begins with base symbols and repeatedly merges adjacent fragments according to learned ranks. WordPiece and unigram/SentencePiece use different training objectives, so a vocabulary alone cannot reproduce their behavior. A large vocabulary shortens common text but enlarges the embedding and output layers; a small vocabulary offers easy coverage but longer sequences. Chinese, code, numbers, and emoji have different frequency structures, so character count cannot predict token count. Engineering comparisons should use the product corpus's length distribution, encoding time, and model quality—not one English sentence.
Treat special tokens and the chat template as one indivisible ABI
A chat model is trained on a token sequence rendered from role/content messages by a template, not on the application's object array. The template defines system, user, and assistant boundaries, whether BOS/EOS tokens are inserted, and where the generation prompt ends. When text is rendered first and tokenized afterward, duplicate special-token insertion should normally be disabled. Switching models means switching the tokenizer and template together. User text must also be distinguished from permitted control tokens so ordinary content cannot accidentally cross message boundaries.
Budget the encoding path as well as the context limit
Count tokens only after applying the template, including the system prompt, history, tool descriptions, and reserved output; character count is not a substitute. Re-encoding the entire text after each character is appended on an edge device creates quadratic repeated work. A stable prefix can be cached, or an incremental implementation used, but merges may cross the append boundary, so two token lists cannot be concatenated naively. Truncate long inputs by message or semantic block and regenerate the template; never cut through a UTF-8 byte sequence or special token.
Use golden vectors to verify consistency across languages and implementations
Build a small corpus containing Chinese, English, spaces, line breaks, emoji, combining characters, code, and text resembling special tokens. Save the tokenizer hash, template version, expected IDs, and decoded result. Compare each case across Python, the host runtime, and the device implementation, and verify the documented encode→decode reversibility boundary. If IDs change after a library upgrade—even if decoded text looks identical—an old prompt/KV cache cannot safely be reused. Cache keys must include the complete tokenizer contract.
Find the first wrong token through layered snapshots
When debugging token differences, do not print only final IDs. Save the source bytes and Unicode code points, then snapshot the normalized text, rendered template, pre-tokenized pieces, and every token ID. Align on the first divergence: if strings differ, inspect line endings, Jinja whitespace control, and BOS/EOS; if strings match but pieces differ, inspect tokenizer.json and merge ranks; if IDs match but model behavior differs, inspect positions and the attention mask. This localizes the fault to a specific protocol layer.
Interactive process
How a conversation becomes a train of integers
The step-by-step player shows what each protocol layer adds or changes, then reconstructs text incrementally from token fragments.
Organize messages · [{role, content}, …]
Preserve the structure of roles, turns, and tool fields
The object structure is not yet model input
Loop / return condition: A new turn returns to “Organize messages” and reapplies the template. Reusing a prefix requires the template, tokenizer hash, token sequence, and positions all to match.
View the complete static diagram
UTF-8 text → normalize → tokenize → [BOS, 1203, 88, EOS] → embedding lookup
Code or command example
ids = tokenizer.encode("temperature 28°C")
print(ids, len(ids))
prompt = template(system, user)
# token budget = prompt_tokens + generated_tokensHands-on lab
Lab notes and export
Engineering pitfall
Knowledge check
Answer all three multiple-choice questions, then submit. Answers stay hidden until submission.
Discuss this chapter on GitHub
Sign in with GitHub to ask a question, share measurements, or compare implementations. Comments are stored in this course's GitHub Discussions.
The embedded comments need JavaScript. You can also open the GitHub discussion area directly: Open GitHub Discussions ↗
Further reading
Bridge to the next chapter
Next, the Transformer becomes a standalone architecture: tokens, positions, blocks, FFNs, residuals, normalization, and logits form one complete path.