Tokens, Tokenisers and Vocabulary

By the end of this lesson you will be able to say why models work on tokens rather than words or characters; train a byte-pair-encoding vocabulary by hand and with the tokenizers library and get the same merges both ways; open a real tokenizer.json, replay its merge rules on a word and predict the pieces before you run it; explain with numbers why one sentence costs 14 tokens in English and 25 in Portuguese; turn a context window into a budget with the template overhead and the answer reserve subtracted; and recognise the special tokens that a chat template puts around your messages. Every cost figure in this course, context length, memory for the KV cache, price per request against a hosted API, is denominated in tokens, so this is the unit the rest of the course counts in.

Every snippet here runs on the CPU in seconds. bpe-by-hand.py and byte-to-char.py need nothing beyond the Part 1 environment; replay-merges.py needs only the Qwen3-1.7B (Apache-2.0) download from this part’s lab (its task 2); chat-template-cost.py and tokenizers-offsets.py need that download and the transformers and tokenizers libraries the lab’s Requirements install; train-tiny-bpe.py needs only the tokenizers library. The outputs printed below were produced with transformers transformers 5.16.1 · verified 2026-09-08 and tokenizers 0.23.2 against the Qwen3-0.6B (Apache-2.0) directory, whose tokeniser files are the same bytes as Qwen3-1.7B’s and Qwen3-8B’s: the Hub file listings of all three repositories give tokenizer.json the same SHA-256, aeb13307…, and the same object ids for tokenizer_config.json, vocab.json and merges.txt, checked 2026-09-12.

Why not words, and why not characters

The Transformers tokenisation summary sets out both failures. Word-level tokenisation splits on spaces and punctuation, and then “vocabulary size becomes extremely large because every unique word requires its own token, including all variations”, which makes the embedding matrix enormous. Worse, “words not in the vocabulary map to an <unk> token, so the model can’t handle new words”. A misspelling, a product name or a new library becomes a hole in the input.

Character-level tokenisation has the opposite problem. The vocabulary is tiny and nothing is unknown, “but sequences become much longer” and “a character like l carries far less meaning than love, so performance suffers”. Longer sequences are not cosmetic: the attention lesson shows that cost grows faster than length.

Subword tokenisation sits between the two. Common words stay whole, rare ones break into pieces the model has seen before, and the vocabulary is a fixed size chosen before training. The documentation gives annoyingly as the example, which “might be split into ["annoying", "ly"] or ["annoy", "ing", "ly"] depending on the vocabulary”. One sentence under the three schemes:

Scheme	Vocabulary	“The quick brown fox jumps over the lazy dog near the river bank.”	Input outside the vocabulary
Word-level	One entry per distinct word form; unbounded	13 tokens	Becomes <unk>; the model receives nothing
Character-level	The alphabet; tens to thousands of entries	64 tokens	Nothing is unknown
Byte-level BPE, Qwen3	151,643 learned entries	14 tokens, measured below	Falls back to bytes: more tokens, never <unk>

Byte-pair encoding, learned from data

Byte-pair encoding (BPE) is, in the documentation’s words, “the most popular tokenization algorithm in Transformers”, used by the Llama, Gemma and Qwen families. It is a training procedure: start from a base alphabet, count every adjacent pair of symbols in the corpus, merge the most frequent pair into one new symbol, record that merge as a rule, and repeat until the vocabulary reaches the size you chose. It produces two things, a vocabulary (the base alphabet plus one entry per merge) and an ordered list of merge rules, and encoding new text replays the rules in the order they were learned.

The documentation’s example is small enough to follow to the last digit. The corpus is five words with frequencies, ("hug", 10), ("pug", 5), ("pun", 12), ("bun", 4), ("hugs", 5), and the base vocabulary is their characters, ["b", "g", "h", "n", "p", "s", "u"].

Learning a byte-pair-encoding vocabulary

1. 1Split into characters("h" "u" "g", 10), ("p" "u" "g", 5), ("p" "u" "n", 12), ("b" "u" "n", 4), ("h" "u" "g" "s", 5)
2. 2Merge the most frequent pair"u"+"g" appears in hug, pug and hugs, so "ug" joins the vocabulary.
3. 3Merge the next one"u"+"n" appears in pun and bun, so "un" joins the vocabulary.
4. 4Repeat to the target sizeThe final size is the base vocabulary plus the number of merges.

The worked example from the Transformers tokenisation summary. Each merge adds one entry to the vocabulary and one rule to an ordered list; encoding new text replays the rules in the same order. Training stops when the vocabulary reaches its target size.

Every pair count is weighted by the frequency of the word it occurs in, and a merge changes the counts of its neighbours, which is why the ranking shifts between steps:

Step	Pair counts, weighted by word frequency	Merge	Vocabulary size
1	ug 20, pu 17, un 16, hu 15, gs 5, bu 4	u+g → ug	8
2	un 16, hug 15, pu 12, pug 5, ugs 5, bu 4	u+n → un	9
3	hug 15, pun 12, pug 5, ugs 5, bun 4	h+ug → hug	10

pu falls from 17 to 12 after the first merge because pug no longer contains a p–u pair once ug is one symbol; only the twelve pun still do. This is the whole algorithm, and it fits in forty lines of standard-library Python:

RunnableAll tracks

bpe-by-hand.py - learn three merges from the documentation's corpus, then encode with them

"""Learn a byte-pair-encoding vocabulary from the five-word corpus in the Transformers
tokenisation summary, printing every pair count, then encode new words by replaying the merges."""
from collections import Counter

corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5}
words = {tuple(w): n for w, n in corpus.items()}      # each word as a tuple of symbols
vocab = sorted({c for w in words for c in w})          # base vocabulary: the characters
merges = []                                            # the ordered merge rules

def merge_pair(w, a, b):
    out, i = [], 0
    while i < len(w):
        if i + 1 < len(w) and (w[i], w[i + 1]) == (a, b):
            out.append(a + b); i += 2
        else:
            out.append(w[i]); i += 1
    return tuple(out)

for step in range(1, 4):                               # three merges, as in the documentation
    pairs = Counter()
    for w, n in words.items():
        for a, b in zip(w, w[1:]):
            pairs[(a, b)] += n                         # weighted by the word's frequency
    (a, b), count = pairs.most_common(1)[0]
    print(f"step {step}  counts  " + ", ".join(f"{x}{y}:{c}" for (x, y), c in pairs.most_common()))
    print(f"step {step}  merge {a!r}+{b!r} ({count} occurrences) -> {a + b!r}")
    merges.append((a, b))
    vocab.append(a + b)
    words = {merge_pair(w, a, b): n for w, n in words.items()}

print(f"vocabulary ({len(vocab)} entries): {vocab}")
print(f"merge rules, in order: {merges}")

def encode(word):
    symbols = [c if c in vocab else "<unk>" for c in word]
    for a, b in merges:                                # replay the rules in training order
        symbols = list(merge_pair(tuple(symbols), a, b))
    return symbols

for word in ["hug", "bug", "mug", "pugs"]:
    print(f"encode {word!r:7} -> {encode(word)}")

Output — what you should see

step 1  counts  ug:20, pu:17, un:16, hu:15, gs:5, bu:4
step 1  merge 'u'+'g' (20 occurrences) -> 'ug'
step 2  counts  un:16, hug:15, pu:12, pug:5, ugs:5, bu:4
step 2  merge 'u'+'n' (16 occurrences) -> 'un'
step 3  counts  hug:15, pun:12, pug:5, ugs:5, bun:4
step 3  merge 'h'+'ug' (15 occurrences) -> 'hug'
vocabulary (10 entries): ['b', 'g', 'h', 'n', 'p', 's', 'u', 'ug', 'un', 'hug']
merge rules, in order: [('u', 'g'), ('u', 'n'), ('h', 'ug')]
encode 'hug'   -> ['hug']
encode 'bug'   -> ['b', 'ug']
encode 'mug'   -> ['<unk>', 'ug']
encode 'pugs'  -> ['p', 'ug', 's']

The encoder never counts anything: it replays the three rules in order, so bug becomes b, ug and mug becomes <unk>, ug because m was not in the base alphabet. pugs stays three pieces because no rule ever joined p to ug or ug to s. Two properties of this procedure explain most of what you will see when you print token boundaries. Frequent whole words survive as single tokens because they were merged early. Anything rare is left as the fragments the merges happened to produce, which is why an unusual identifier in source code can cost five or six tokens while a common English word costs one.

Vocabulary sizes are a design choice made before training:

Tokeniser	Base alphabet	Merges and additions	Entries	Source
GPT	478 characters	40,000 merges	40,478	Transformers summary
GPT-2	256 byte values	50,000 merges + 1 end-of-text token	50,257	Transformers summary
Qwen3 (all sizes)	256 byte values	151,387 merges + 26 added tokens	151,669 in the tokeniser; 151,936 embedding rows in config.json	tokenizer.json, config.json
Llama 3 and 3.1	256 byte values	“100K tokens from the tiktoken tokenizer with 28K additional tokens to better support non-English languages”	“128K” in the paper, plus 256 reserved special tokens in tokenizer.py	Llama 3 paper §3.2; llama-models

The Qwen3 row is where the width of the output distribution in the previous lesson came from, and the gap between 151,669 and 151,936 is padding to a multiple the matrix kernels like; the lab reconciles those three numbers.

Byte-level BPE

Unicode has far too many characters to use as a base alphabet. The documented fix is byte-level BPE, which “uses 256 byte values as the base vocabulary instead, ensuring every word can be tokenized without the <unk> token”. The mechanism is a fixed display table, published in GPT-2’s encoder.py and used unchanged by Qwen3: the 188 byte values that are printable on their own are shown as themselves, and the other 68 (the controls, space, DEL, the C1 range, no-break space and soft hyphen) are given the code points from U+0100 upward, in byte order. BPE then merges those characters exactly as it merged u and g above.

RunnableAll tracks

byte-to-char.py - the 256-entry display table behind Ġ, Ċ and á

"""The display alphabet of byte-level BPE: each of the 256 byte values becomes one printable
character, so that spaces, newlines and non-ASCII bytes can be merged like any other symbol."""
def bytes_to_unicode():
    keep = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1))
    byte_values, chars, n = keep[:], keep[:], 0
    for b in range(256):
        if b not in keep:                  # the 68 bytes that are not printable on their own
            byte_values.append(b)
            chars.append(256 + n)          # get the code points U+0100 onward, in byte order
            n += 1
    return dict(zip(byte_values, map(chr, chars)))

table = bytes_to_unicode()
for text in [" ", "\n", "\t", "a", "á", "ç", "🚀"]:
    shown = "".join(table[b] for b in text.encode("utf-8"))
    print(f"{text!r:8} utf-8 {text.encode('utf-8').hex(' '):12} shown as {shown!r}")

Output — what you should see

' '      utf-8 20           shown as 'Ġ'
'\n'     utf-8 0a           shown as 'Ċ'
'\t'     utf-8 09           shown as 'ĉ'
'a'      utf-8 61           shown as 'a'
'á'      utf-8 c3 a1        shown as 'á'
'ç'      utf-8 c3 a7        shown as 'ç'
'🚀'      utf-8 f0 9f 9a 80  shown as 'ðŁļĢ'

Space is byte 0x20, the 33rd non-printable byte, so it becomes U+0120, Ġ; newline is Ċ. That is all Ġworld means: the token for a space followed by world. Non-ASCII text is spent one byte at a time unless merges have joined the bytes: á is two bytes, and Qwen3’s rule 1697 joins Ã+¡ back into one symbol; the rocket is four bytes and one token on its own, id 145836, but three tokens after a space, ĠðŁ, ļ, Ģ: the pre-tokeniser hands BPE the space and all four bytes as one chunk, the rule Ġ+ðŁ (rank 10906) fires before ðŁ+ļ (rank 123590) can, and no rule joins ĠðŁ to what follows. Run replay-merges.py below on " 🚀" and on "🚀" to watch both paths. Strange input never stops a byte-level tokeniser; it just costs more.

The pre-tokeniser decides what may be merged

Before any merge runs, a regular expression cuts the text into chunks, and BPE works inside a chunk only. This is the first thing to read in a tokenizer.json, because it fixes behaviour that no amount of vocabulary can change. Qwen3’s pattern, from the pre_tokenizer entry of the file:

(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

Read left to right: English contractions are split off; one optional non-letter (usually a space) plus a run of letters is a chunk, which is why the space belongs to the word that follows it; a single digit is a chunk on its own; a run of punctuation with an optional leading space is a chunk; then rules for runs of whitespace and newlines. The consequences, all measured with the Qwen3 tokeniser:

Text	Pieces	Ids	What the pattern did
the	the	1782	A word with no space before it
the	Ġthe	279	A different token: the space is part of it
1969	1, 9, 6, 9	16, 24, 21, 24	\p{N} isolates every digit before BPE sees them
2026-09-12	ten single characters		Digits and punctuation never merge across the boundary
I'm	I, 'm	40, 2776	The contraction alternative fires first
):	):	1648	A punctuation run is one chunk, and this one had a merge
four spaces	ĠĠĠĠ	257	Whitespace runs are chunks; the first two rules in merges.txt, Ġ Ġ (rank 0, applied twice) and ĠĠ ĠĠ (rank 1), joined them

Llama 3’s pattern in tokenizer.py, the family that Llama 3.1 8B Instruct (Llama 3.1 Community licence, gated on the Hub) belongs to, differs from this in one place, \p{N}{1,3} instead of \p{N}, so 1969 reaches its BPE as the chunks 196 and 9; whether 196 is then one token depends on that model’s vocabulary, which this course cannot open without accepting that licence. Either way, the model receives digits as pieces whose boundaries a regular expression chose, not as a number, and that is the mechanical reason arithmetic on long numbers is fragile and why a model cannot count the letters in a word it received as token, isation: the letters were never separately present in the input. Neither is a property of the architecture, and both are worth checking before concluding from a failed puzzle that a model is worse than it is.

SentencePiece, Unigram and WordPiece

Plain BPE begins by splitting on whitespace, which assumes the language uses it. SentencePiece, which its README describes as “a fast, lightweight, and unsupervised text tokenizer and detokenizer designed for neural network-based text generation systems”, removes that assumption. It treats the input “as a raw sequence of Unicode characters” and, in the README’s words, “escapes whitespaces with a meta-symbol ▁ (U+2581) and includes it in the tokenization. This design ensures that detokenization is a simple, lossless string join operation”. Chinese, Japanese and Thai then need no language-specific pre-tokeniser, and ▁the and the are different tokens for the same reason Ġthe and the are.

SentencePiece can run BPE or the Unigram algorithm underneath. Unigram starts from a large candidate vocabulary and removes the entries whose removal costs the least likelihood, which makes it probabilistic rather than rule-replaying: “Unigram picks the highest probability tokenization” among several possible splits of the same word. WordPiece, the BERT-family algorithm, merges upward like BPE but scores a pair by its joint frequency divided by the product of the parts’ frequencies; “BPE simply merges whichever pair appears the most. WordPiece measures how informative each merge is.” You will not meet WordPiece on a generative model in this course, but you will on some embedding and reranking models, which is why their boundaries look different.

Algorithm	Direction	Chooses by	Space handling	Used by
BPE, byte-level	Merge upward	Highest pair frequency	Byte 0x20 shown as Ġ	GPT-2, Llama, Qwen, Gemma (per the summary)
BPE or Unigram via SentencePiece	Up, or prune downward	Frequency, or likelihood loss	▁ in the vocabulary	T5 and other SentencePiece models
WordPiece	Merge upward	Frequency ÷ product of part frequencies	## marks a continuation	BERT, DistilBERT, Electra

Tokens, words and bytes are three different counts

A token is not a word and not a byte, and the ratios between them are properties of a particular tokeniser and a particular text. The same sentence, or as close as translation allows, in fourteen forms through the Qwen3 tokeniser (words are whitespace-separated; Chinese and Japanese have none):

Sample	Characters	UTF-8 bytes	Words	Tokens	Bytes per token	Tokens per word
English	64	64	13	14	4.57	1.08
Portuguese	77	80	14	25	3.20	1.79
Spanish	80	83	15	25	3.32	1.67
French	74	75	13	23	3.26	1.77
German	68	69	11	20	3.45	1.82
Russian	67	124	10	29	4.28	2.90
Arabic	62	113	11	25	4.52	2.27
Hindi	59	155	12	58	2.67	4.83
Chinese	17	51	1	12	4.25
Japanese	26	78	1	21	3.71
Python signature and body	87	87	10	24	3.62	2.40
JSON object	71	71	9	29	2.45	3.22
1969 2026-09-12 3.14159 1,000,000	33	33	4	33	1.00	8.25
Four emoji	8	22	4	10	2.20	2.50

Four things to read out of it. English is cheap because the merge corpus was English-heavy, so its common words are single tokens and it runs at over four bytes per token. The Romance languages cost about 1.8 tokens per word: rápida survived as one token (merge rule 135,034 of 151,387, so a late and rare one), but castanha, preguiçoso and perto did not. Hindi is the expensive case, 58 tokens for one sentence, because Devanagari is three bytes per character and few of its byte sequences were merged. Numbers cost exactly one token per character by construction of the pre-tokeniser, and JSON is worse than prose because of its punctuation and digits. Compare languages by bytes per token, not by raw counts, and expect a document’s token count to be roughly its UTF-8 size divided by the figure in that column for its language.

To see exactly why a word ends up as the pieces it does, replay the model’s own rules. Every entry in tokenizer.json’s merges list has a rank, its position in the list, and encoding always fires the lowest-ranked rule that applies:

RunnableAll tracks

replay-merges.py - fire a model's merge rules on one word, lowest rank first

"""Replay a model's own merge rules on one word, printing which rule fires at each step and
its rank in merges.txt, to see exactly why a word ends up as the pieces it does.
Usage: python replay-merges.py [model-dir] [word]      (the word may start with a space)"""
import json
import sys
from pathlib import Path

model_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "~/llm-course/models/qwen3-1.7b").expanduser()
word = sys.argv[2] if len(sys.argv) > 2 else " castanha"

tok = json.loads((model_dir / "tokenizer.json").read_text(encoding="utf-8"))
vocab = tok["model"]["vocab"]                          # token string -> id
rank = {tuple(m): i for i, m in enumerate(tok["model"]["merges"])}   # earlier merge = lower rank

def bytes_to_unicode():
    keep = list(range(33, 127)) + list(range(161, 173)) + list(range(174, 256))
    byte_values, chars, n = keep[:], keep[:], 0
    for b in range(256):
        if b not in keep:
            byte_values.append(b); chars.append(256 + n); n += 1
    return dict(zip(byte_values, map(chr, chars)))

symbols = [bytes_to_unicode()[b] for b in word.encode("utf-8")]
print(f"{word!r} as {len(symbols)} byte symbols: {symbols}")
step = 0
while True:
    pairs = [(rank.get(p, float("inf")), i) for i, p in enumerate(zip(symbols, symbols[1:]))]
    if not pairs or min(pairs)[0] == float("inf"):
        break                                          # no adjacent pair has a merge rule left
    best_rank, i = min(pairs)                          # the lowest rank fires first
    step += 1
    a, b = symbols[i], symbols[i + 1]
    symbols[i:i + 2] = [a + b]
    print(f"step {step}  merge {a!r} + {b!r} -> {a + b!r:12} rank {best_rank:6d}   now {symbols}")
print(f"final pieces {symbols}  ids {[vocab[s] for s in symbols]}")

Output — what you should see

' castanha' as 9 byte symbols: ['Ġ', 'c', 'a', 's', 't', 'a', 'n', 'h', 'a']
step 1  merge 's' + 't' -> 'st'         rank     11   now ['Ġ', 'c', 'a', 'st', 'a', 'n', 'h', 'a']
step 2  merge 'Ġ' + 'c' -> 'Ġc'         rank     16   now ['Ġc', 'a', 'st', 'a', 'n', 'h', 'a']
step 3  merge 'a' + 'n' -> 'an'         rank     20   now ['Ġc', 'a', 'st', 'an', 'h', 'a']
step 4  merge 'a' + 'st' -> 'ast'        rank    303   now ['Ġc', 'ast', 'an', 'h', 'a']
step 5  merge 'h' + 'a' -> 'ha'         rank   3967   now ['Ġc', 'ast', 'an', 'ha']
step 6  merge 'Ġc' + 'ast' -> 'Ġcast'      rank   6055   now ['Ġcast', 'an', 'ha']
final pieces ['Ġcast', 'an', 'ha']  ids [6311, 276, 4223]

Six rules fire and then none applies: there is no rule joining Ġcast to an or an to ha, because that sequence was never frequent enough in the corpus to earn one, so the Portuguese word for chestnut arrives as three pieces the model has seen in many other words. Run it on " the" and three rules of rank 3, 14 and 23 produce Ġthe, id 279, in three steps: the earliest merges in the file are the most frequent things in the corpus. Run python replay-merges.py ~/llm-course/models/qwen3-1.7b " " and the first two rules in the file, rank 0 twice and rank 1 once, turn four spaces into the single token 257 from the pre-tokeniser table. The script reproduces the real tokeniser’s ids for every one of 11,230 chunks of mixed text it was checked against; the replay is the algorithm, not an approximation of it.

▷Predict, then measure

Before the lab, write down your guess for how many tokens each of these costs with the Qwen3 tokeniser: Hello, world!; the same sentence in Portuguese; def get_user_by_id(user_id: int):; and 🇧🇷🚀. Then run the lab’s tokenising script with your four numbers and compare. The code and emoji lines are the ones to check most carefully; the lab’s script prints your guess and the measured count side by side, and the point is to replace an intuition with a measurement you made.

The count depends on the model as much as the language

Two models with different tokenisers give different counts for the same text, and the difference comes from three places you can now name: the pre-tokeniser pattern, the merge list and its corpus, and the number and names of the special tokens.

	Qwen3 family	Llama 3.1
Implementation	tokenizers BPE model, ByteLevel pre-tokeniser and decoder	tiktoken-style BPE, per tokenizer.py
Vocabulary	151,643 learned + 26 added	“128K” learned, 256 reserved special
Digits	One digit per chunk	Up to three digits per chunk
Turn markers	<|im_start|>, <|im_end|>	<|start_header_id|>, <|end_header_id|>, <|eot_id|>
Sequence start	none added; add_bos_token is false	<|begin_of_text|>
English compression reported	not stated on the card	“3.94 characters per token” on the paper’s English sample

The rule that follows is short: count tokens with the tokeniser of the model that will serve the request, and treat a count made with any other tokeniser as an estimate to check, not a number to budget with. The Llama paper’s own figure shows how much a tokeniser change moves the count: the same English sample went from 3.17 to 3.94 characters per token between Llama 2 and Llama 3, a fifth fewer tokens for the same text.

Token budgets and the context window

The context window is stated in tokens, and everything that has to fit inside it competes for the same budget:

prompt_tokens = template_overhead + system + tool_definitions + retrieved_documents
              + conversation_so_far + new_message
prompt_tokens + reserved_for_answer  <=  context_length

context_length is what you ask the engine to allocate, at most the window the model was trained for, which the Qwen3-8B card gives as 32,768 tokens natively and “up to 131,072 tokens” with YaRN scaling. The template overhead is measurable, and for Qwen3 it is small per turn but not zero:

Item, Qwen3 chat template	Tokens	Measured how
Framing of one user or system message, <|im_start|>user, newline, <|im_end|>, newline	5	One-character message rendered: 6 tokens
Generation prompt, <|im_start|>assistant, newline	3	add_generation_prompt=True
Empty thinking block when enable_thinking=False	4	<think>, two newlines, </think>, two newlines
Tools section with one small function definition	147	chat-template-cost.py in the next section: a one-character request costs 160 tokens with tools= against 13 without; the definition’s JSON alone is 64, and the system turn, instructions and <tools> tags the template wraps around it 83

Arithmetic from stated inputs, not a measurement, for one request to Qwen3-8B at its native context:

Component	Tokens	How it was counted
Context length	32,768	The card’s native window
System prompt	300	Counted with the tokeniser
Eight tool definitions	147 + 7 × 65 = 602	The measured block, plus one JSON and newline per further tool of the same size
Six retrieved chunks of 512 tokens	3,072	Chunk size chosen when indexing
Twenty earlier turns of 200 tokens plus 5 framing	4,100	Conversation so far
Reserved for the answer	2,048	Whatever max_tokens you will ask for
Left for the new message	32,768 − 300 − 602 − 3,072 − 4,100 − 2,048 = 22,646	The remainder

Now change the language. A 20,000-token English document, at the sentence-level ratio measured above, is about 20,000 × 25 ÷ 14 = 35,700 tokens in Portuguese, more than the whole native window on its own, and about 82,900 in Hindi. The parameters lesson turns those tokens into bytes of KV cache; with the per-token figure the course records for Qwen3-8B in FP16, 147,456 bytes, the same document costs about 2.95 GB of cache in English and 5.3 GB in Portuguese, again arithmetic rather than measurement.

▤Memory budget

When a request is close to a context limit, the fix is to send fewer tokens, not to change the model. Trimming a retrieved chunk, dropping stale turns or removing an unused tool definition frees exactly as many tokens as it cost, and the table above says which line is worth the most. What happens when the sum exceeds the context length depends on the engine, a refused request or silent truncation of the oldest tokens; check the serving engine’s documentation for which before relying on either.

Special tokens are ordinary vocabulary entries

A tokeniser’s vocabulary contains more than fragments of text. Qwen3’s tokenizer.json lists 26 added tokens after the 151,643 learned entries, ids 151643 to 151668, and config.json names two of them: bos_token_id 151643 and eos_token_id 151645. The roles below are read from the token names and from tokenizer_config.json; Qwen publishes no document defining each one.

Ids	Tokens	Role
151643	<|endoftext|>	End of a document in pretraining; the padding token; the base checkpoint’s end-of-sequence
151644, 151645	<|im_start|>, <|im_end|>	Start of a turn; end of a turn, the instruct checkpoint’s end-of-sequence
151646 to 151656	<|object_ref_start|> … <|video_pad|>	Vision and grounding markers shared with the multimodal family
151657, 151658	<tool_call>, </tool_call>	Around a function call the model emits
151659 to 151664	<|fim_prefix|> … <|file_sep|>	Fill-in-the-middle and repository markers for code
151665, 151666	<tool_response>, </tool_response>	Around a tool result sent back
151667, 151668	<think>, </think>	The thinking block

They are not magic. They are rows of the same embedding matrix, learned the same way, and the model was trained on text in which they appeared in specific places. The chat template, a Jinja string stored in tokenizer_config.json, is what puts them there. Render one and count:

RunnableAll tracks

chat-template-cost.py - render a conversation and count what the template added, then add one tool

"""Render a conversation through the model's own chat template and count what the template adds,
then add one tool definition and count what that adds.
Usage: python chat-template-cost.py [model-dir]"""
import json
import sys
from transformers import AutoTokenizer

model_dir = sys.argv[1] if len(sys.argv) > 1 else "~/llm-course/models/qwen3-1.7b"
tok = AutoTokenizer.from_pretrained(model_dir)
def count(text):
    return len(tok(text, add_special_tokens=False)["input_ids"])

messages = [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user", "content": "How many tokens is this?"},
]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
print(text)
print("---")
ids = tok(text, add_special_tokens=False)["input_ids"]
print("pieces:", tok.convert_ids_to_tokens(ids))
body = sum(count(m["content"]) for m in messages)
print(f"tokens in the rendered prompt {len(ids)}, in the two message bodies {body}, added by the template {len(ids) - body}")
default = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(f"with enable_thinking left at its default the prompt ends {default[-22:]!r} and costs {count(default)} tokens")

tools = [{"type": "function", "function": {"name": "get_weather", "description": "Get the current weather for a city.",
          "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "City name"}},
                         "required": ["city"]}}}]
one = [{"role": "user", "content": "x"}]
plain = count(tok.apply_chat_template(one, tokenize=False, add_generation_prompt=True, enable_thinking=False))
with_tool = count(tok.apply_chat_template(one, tools=tools, tokenize=False, add_generation_prompt=True, enable_thinking=False))
print(f"one-character request {plain} tokens; with one tool definition {with_tool}; the tool's JSON alone "
      f"{count(json.dumps(tools[0]))}; the system turn, instructions and <tools> tags around it {with_tool - plain - count(json.dumps(tools[0]))}")

Output — what you should see

<|im_start|>system
You are a terse assistant.<|im_end|>
<|im_start|>user
How many tokens is this?<|im_end|>
<|im_start|>assistant
<think>

</think>


---
pieces: ['<|im_start|>', 'system', 'Ċ', 'You', 'Ġare', 'Ġa', 'Ġterse', 'Ġassistant', '.', '<|im_end|>', 'Ċ', '<|im_start|>', 'user', 'Ċ', 'How', 'Ġmany', 'Ġtokens', 'Ġis', 'Ġthis', '?', '<|im_end|>', 'Ċ', '<|im_start|>', 'assistant', 'Ċ', '<think>', 'ĊĊ', '</think>', 'ĊĊ']
tokens in the rendered prompt 29, in the two message bodies 12, added by the template 17
with enable_thinking left at its default the prompt ends '<|im_start|>assistant\n' and costs 25 tokens
one-character request 13 tokens; with one tool definition 160; the tool's JSON alone 64; the system turn, instructions and <tools> tags around it 83

Twelve tokens of message, seventeen of structure: five framing each turn, three for the generation prompt that the documentation says “adds tokens to the end of the chat that indicate the start of an assistant response”, and four for the empty thinking block the template inserts when enable_thinking is false, which is how the Qwen3 card says non-thinking mode is selected. Pass tokenize=True instead and the same call returns the ids directly. The last line is where the 147 in the budget table came from: the documentation’s tools= argument takes a list of JSON-schema function definitions, and Qwen3’s template answers one by opening a system turn that holds the instructions, the <tools> tags and the definition itself, 83 tokens plus the 64 of the JSON.

The chat-template documentation carries two warnings that matter here. First, “chat templates should already include all the necessary special tokens, and adding additional special tokens is often incorrect or duplicated, hurting model performance. When you format text with apply_chat_template(tokenize=False), make sure you set add_special_tokens=False if you tokenize later”. Qwen3 adds no beginning-of-sequence token, so on this model the default is harmless; on a model that does, the prompt gains a second one the model never saw in training. Second, the same conversation rendered by two models fine-tuned from the same base looks nothing alike, one using [INST] markers and the other <|user|> and <|assistant|>, and “with the wrong control tokens, these models would have drastically worse performance”. Part 6 and Part 7 return to this when engines apply templates on your behalf, and Part 11 uses the same call with add_generation_prompt=False to build training data.

✕Common mistake

Control tokens are matched as text. Tokenising the string <|im_start|>user typed inside a user message gives [151644, 872]: the tokeniser recognised the added token, so user-supplied text can close or open a turn from inside the message. Passing split_special_tokens=True to the tokeniser call returns seven ordinary tokens instead. Whether a serving layer does that for you is a per-engine question; Part 26 returns to prompt injection generally. A second failure you will meet on a bare Transformers install: apply_chat_template renders with Jinja and fails with ImportError: apply_chat_template requires jinja2 to be installed if nothing pulled jinja2 in; PyTorch depends on it, so the Part 1 environment already has it.

The tokenizers library, hands-on

The Hugging Face tokenizers library is the implementation underneath AutoTokenizer in the lab; this section calls it directly. Its quicktour describes it as providing “an implementation of today’s most used tokenizers that is both easy to use and blazing fast”, implemented in Rust, and lists the property that matters when debugging: “full alignment tracking, meaning you can always get the part of your original sentence that corresponds to a given token”. The API reference states the pipeline that a Tokenizer runs: a normaliser, a pre-tokeniser, the model, and a post-processor; a decoder reverses it. Every stage is data in tokenizer.json:

RunnableAll tracks

tokenizers-offsets.py - the pipeline stages, and every token mapped back to its characters

"""Use the tokenizers library directly on a model's tokenizer.json: the pipeline stages, and
the offsets that map every token back to the characters it came from.
Usage: python tokenizers-offsets.py [model-dir]"""
import sys
from pathlib import Path
from tokenizers import Tokenizer

model_dir = Path(sys.argv[1] if len(sys.argv) > 1 else "~/llm-course/models/qwen3-1.7b").expanduser()
tokenizer = Tokenizer.from_file(str(model_dir / "tokenizer.json"))
for stage in ("normalizer", "pre_tokenizer", "model", "post_processor", "decoder"):
    print(f"{stage:14} {type(getattr(tokenizer, stage)).__name__}")
print(f"vocabulary     {tokenizer.get_vocab_size(with_added_tokens=False):,} merged entries, "
      f"{tokenizer.get_vocab_size(with_added_tokens=True):,} with added tokens")

text = "A raposa preguiçosa: 1969 🚀"
print("pre-tokenised:", [piece for piece, _ in tokenizer.pre_tokenizer.pre_tokenize_str(text)])
enc = tokenizer.encode(text, add_special_tokens=False)
print(f"{'id':>7}  {'token':14} {'offsets':10} text")
for i, t, (a, b) in zip(enc.ids, enc.tokens, enc.offsets):
    print(f"{i:7d}  {t!r:14} {str((a, b)):10} {text[a:b]!r}")
print("decode(ids) ==", repr(tokenizer.decode(enc.ids)))
print("token_to_id('<|im_start|>') =", tokenizer.token_to_id("<|im_start|>"), "  id_to_token(279) =", repr(tokenizer.id_to_token(279)))

Output — what you should see

normalizer     NFC
pre_tokenizer  Sequence
model          BPE
post_processor ByteLevel
decoder        ByteLevel
vocabulary     151,643 merged entries, 151,669 with added tokens
pre-tokenised: ['A', 'Ġraposa', 'Ġpreguiçosa', ':', 'Ġ', '1', '9', '6', '9', 'ĠðŁļĢ']
     id  token          offsets    text
     32  'A'            (0, 1)     'A'
   7327  'Ġrap'         (1, 5)     ' rap'
  11983  'osa'          (5, 8)     'osa'
    855  'Ġpre'         (8, 12)    ' pre'
  19109  'gui'          (12, 15)   'gui'
   3131  'ç'           (15, 16)   'ç'
  11983  'osa'          (16, 19)   'osa'
     25  ':'            (19, 20)   ':'
    220  'Ġ'            (20, 21)   ' '
     16  '1'            (21, 22)   '1'
     24  '9'            (22, 23)   '9'
     21  '6'            (23, 24)   '6'
     24  '9'            (24, 25)   '9'
  11162  'ĠðŁ'          (25, 27)   ' 🚀'
    248  'ļ'            (26, 27)   '🚀'
    222  'Ģ'            (26, 27)   '🚀'
decode(ids) == 'A raposa preguiçosa: 1969 🚀'
token_to_id('<|im_start|>') = 151644   id_to_token(279) = 'Ġthe'

The normaliser is Unicode NFC, so a decomposed ç and a precomposed one become the same bytes before anything else happens. The pre-tokeniser is a Sequence, the regex split above followed by the byte-to-character mapping; you can see its output before the model runs. The offsets are the alignment tracking: osa at (5, 8) and again at (16, 19) is the same id 11983 in two words, and the three tokens that share the rocket all point at character 26, which is how a user interface can highlight the source of any token, including one that holds only part of a character. decode reverses the whole pipeline losslessly.

The same library trains tokenisers, and training it on the documentation’s corpus reproduces the hand trace above rule for rule, with the trainer’s vocab_size counting the special token, the alphabet and the merges together:

RunnableAll tracks

train-tiny-bpe.py - the tokenizers library learns the same three merges

"""Train a byte-pair-encoding tokeniser with the tokenizers library on the five-word corpus from
the Transformers documentation, and compare its merges with the ones worked out by hand."""
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.pre_tokenizers import Whitespace
from tokenizers.trainers import BpeTrainer

corpus = {"hug": 10, "pug": 5, "pun": 12, "bun": 4, "hugs": 5}
lines = [word for word, count in corpus.items() for _ in range(count)]   # each word, count times

tokenizer = Tokenizer(BPE(unk_token="<unk>"))
tokenizer.pre_tokenizer = Whitespace()
trainer = BpeTrainer(vocab_size=11, special_tokens=["<unk>"], show_progress=False)   # <unk> + 7 characters + 3 merges
tokenizer.train_from_iterator(lines, trainer)

print("vocabulary by id:", sorted(tokenizer.get_vocab().items(), key=lambda kv: kv[1]))
for word in ["hug", "bug", "mug", "pugs"]:
    enc = tokenizer.encode(word)
    print(f"encode {word!r:7} -> {enc.tokens}  ids {enc.ids}")
tokenizer.save("tiny-bpe.json")

Output — what you should see

vocabulary by id: [('<unk>', 0), ('b', 1), ('g', 2), ('h', 3), ('n', 4), ('p', 5), ('s', 6), ('u', 7), ('ug', 8), ('un', 9), ('hug', 10)]
encode 'hug'   -> ['hug']  ids [10]
encode 'bug'   -> ['b', 'ug']  ids [1, 8]
encode 'mug'   -> ['<unk>', 'ug']  ids [0, 8]
encode 'pugs'  -> ['p', 'ug', 's']  ids [5, 8, 6]

Open the saved tiny-bpe.json and its merges list reads ["u", "g"], ["u", "n"], ["h", "ug"], the three rules from the table. The full-size tokenizer.json you downloaded is the same structure with 151,387 rules, and vocab.json and merges.txt beside it are the same vocabulary and rules in the older two-file form. In a model repository the tokeniser is data, not code:

File, Qwen3-1.7B	Bytes	Holds
tokenizer.json	11,422,654	Vocabulary, merges, added tokens, the four pipeline stages
tokenizer_config.json	9,732	Special-token names and flags (add_bos_token, eos_token), model_max_length 131072, the chat template
vocab.json	2,776,833	The vocabulary alone, token string to id
merges.txt	1,671,853	The merge rules alone, one per line, in rank order

Which interface to use is a small decision:

You need	Use	Because
The chat template, special-token names, apply_chat_template	transformers.AutoTokenizer	It reads tokenizer_config.json as well as tokenizer.json
Offsets, batch speed, training, or no transformers dependency	tokenizers.Tokenizer.from_file	The Rust pipeline directly, tokenizer.json alone
Tokenisation inside an engine	Nothing; the engine carries it	GGUF embeds the vocabulary and template; Part 6 shows the symptoms of a wrong template and how to print exactly what the server sends

Make a token budget that includes the conversation wrapper

Budget for the serialised request, not just the user’s visible paragraph. A chat template may add role delimiters, a generation prefix and tool definitions. Retrieved passages and earlier tool results can be much larger than the new question. The response allowance must fit alongside all of them within the serving configuration’s effective limit.

For a paper exercise, choose a context capacity of 4,096 tokens. Reserve 768 for the response and 256 as an engineering margin. That leaves 3,072 for the complete input, including template overhead. These are illustrative allocations, not a promise about a particular engine. Count the fully rendered input with the checkpoint’s tokeniser before accepting a request near the boundary.

Repeat the count after adding a tool schema and after replacing an English paragraph with source code. Explain why equal character counts need not mean equal token counts. If the endpoint reports a different count, inspect added templates, special tokens and truncation before blaming the tokeniser. A request accepted by HTTP can still be semantically damaged if important earlier context was removed.

Recap

Models work on subword tokens because word vocabularies are unbounded and character sequences are too long. BPE learns a vocabulary by repeatedly merging the most frequent adjacent pair, records each merge as a ranked rule, and encodes by replaying the rules lowest rank first, inside chunks a pre-tokeniser regular expression chose; byte-level BPE bases the alphabet on the 256 byte values so nothing is ever unknown; SentencePiece puts the space in the vocabulary as ▁, making decoding a lossless join. Tokens, words and bytes are three different counts whose ratio depends on how well the text matches the merge corpus and on which model’s tokeniser you count with. A context window is a budget in which the template, the tools, the documents, the history and the reserved answer all compete. Special tokens are ordinary vocabulary rows in special positions, placed there by the chat template, and using the wrong template is one of the commonest reasons a model suddenly seems much worse.

Check your understanding

Question 1. Why does byte-level byte-pair encoding never need an unknown token?

• Because it stores every Unicode character in the vocabulary
• Because its base vocabulary is the 256 byte values, so any input decomposes into tokens that exist
• Because it replaces unknown characters with a space
• Because it refuses input it cannot encode

Show the answer and why

Answer: Because its base vocabulary is the 256 byte values, so any input decomposes into tokens that exist

The documented design uses 256 byte tokens as the base, plus merges. GPT-2’s vocabulary is 50,257: 256 byte tokens, 50,000 merges and one end-of-text token. Anything at all can be spelled out in bytes, at a cost in token count: Hindi ran at 2.67 bytes per token against 4.57 for English in the table above.

Question 2. In the hand-trained example, why does the count for the pair "p"+"u" drop from 17 to 12 between step 1 and step 2?

• Because "pu" was merged in step 1
• Because merging "u"+"g" removed the "p"–"u" pair from every "pug", leaving only the twelve "pun"
• Because the trainer discards pairs below a frequency threshold
• Because the corpus is re-sampled at every step

Show the answer and why

Answer: Because merging "u"+"g" removed the "p"–"u" pair from every "pug", leaving only the twelve "pun"

After "ug" becomes one symbol, "pug" is the sequence "p", "ug", which has no "p"–"u" pair any more. Pair counts are recomputed after every merge, which is why a merge changes the ranking of its neighbours and why the order of rules matters at encoding time.

Question 3. A request to Qwen3-8B at its native 32,768-token context carries a 300-token system prompt, 602 tokens of tool definitions, 3,072 tokens of retrieved chunks and 4,100 tokens of history, and you reserve 2,048 for the answer. You now attach a 20,000-token English report translated into Portuguese at the ratio measured in this lesson. What happens?

• It fits with about 2,600 tokens to spare
• It does not fit: the report alone is about 35,700 tokens, more than the whole window
• It fits only if enable_thinking is false
• The engine compresses the report to fit

Show the answer and why

Answer: It does not fit: the report alone is about 35,700 tokens, more than the whole window

20,000 × 25 ÷ 14 is about 35,700 tokens before anything else is counted; even the English version, 20,000 against the 22,646 the budget table left, only just fits. The fix is fewer tokens, fewer chunks, fewer turns, or a smaller document, not a different model; and what the engine does when the sum exceeds the context is engine-specific.

Question 4. Which of these lines is the bug in code that prepares a prompt for Qwen3-1.7B? (a) text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) (b) ids = tok(text)["input_ids"] (c) print(len(ids))

• (a): add_generation_prompt must be False when tokenize is False
• (b): the templated string is tokenised with the default add_special_tokens, which the documentation says can duplicate special tokens; pass add_special_tokens=False, or use tokenize=True in (a)
• (c): len counts characters, not tokens
• Nothing is wrong on any model

Show the answer and why

Answer: (b): the templated string is tokenised with the default add_special_tokens, which the documentation says can duplicate special tokens; pass add_special_tokens=False, or use tokenize=True in (a)

The chat-template documentation says to set add_special_tokens=False when you tokenise a string that apply_chat_template already formatted, or to let apply_chat_template tokenise. On Qwen3 the default happens to add nothing because add_bos_token is false, so the bug is silent here and visible on a model that prepends a beginning-of-sequence token; write it correctly everywhere.

Question 5. Which of these are true of special tokens? Select all that apply.

• They are entries in the same vocabulary as ordinary tokens, with rows in the same embedding matrix
• The chat template inserts them; they are not typed by the user
• A control token typed literally inside a user message is still recognised by the tokeniser unless the caller splits special tokens
• They are stored in the model weights rather than in the tokeniser files

Show the answer and why

Answer: They are entries in the same vocabulary as ordinary tokens, with rows in the same embedding matrix, The chat template inserts them; they are not typed by the user, A control token typed literally inside a user message is still recognised by the tokeniser unless the caller splits special tokens

Special tokens live in tokenizer.json and tokenizer_config.json with their ids and names, and the template places them. Because added tokens are matched as text, "<|im_start|>user" inside a message tokenises to [151644, 872] by default, which is why a serving layer has to decide what to do with user-typed control tokens.

Question 6. You print tokens for a sentence and see entries beginning with "▁" from one model and with "Ġ" from another. What are those characters?

• Markers for unknown tokens
• The space that preceded the token: U+2581 in SentencePiece vocabularies, and the display character for byte 0x20 in byte-level BPE
• Markers for the start of a sentence
• Indicators of a byte-level fallback

Show the answer and why

Answer: The space that preceded the token: U+2581 in SentencePiece vocabularies, and the display character for byte 0x20 in byte-level BPE

SentencePiece escapes whitespace as ▁ so that decoding is a lossless join; byte-level BPE shows byte 0x20 as Ġ (U+0120) through the fixed 256-entry display table. In both families a token with the marker and the same token without it are different vocabulary entries with different ids, 279 for "Ġthe" and 1782 for "the" in Qwen3.

Check my answersClear