Watch GPT-2 read.
This is the real GPT-2 tokenizer, embedded in this page. Type anything — text shatters into tokens, and tokens become the integers the model actually sees.
Edit me — the demo types on its own until you take over.
Slice it like GPT-2
Click between characters to place your cuts, then check how the tokenizer really splits it. Five rounds.
Train the vocabulary yourself
BPE starts from single characters and merges the most frequent adjacent pair, over and over. Watch the vocabulary grow on the right.
Press “Merge one pair”, or scroll here and watch it train itself.
Vocabulary · 0 merges
Tokens ≠ words ≠ characters
Three live examples, computed by the embedded tokenizer right now.
Common words stay whole
Frequent words cost one token each — cheap.
Rare words shatter
Long words break into pieces the model has seen before.
其他文字按字节计价
Outside common English, GPT-2 falls back to raw bytes (dashed, shown as hex) — one Chinese character can cost three tokens.
Why any of this matters
Attention, embeddings, KV cache — everything downstream operates on these token IDs, never on your text.
Context windows, latency, and API cost are all measured in tokens — not words, not characters.
Sources
Sennrich, Haddow & Birch — Neural Machine Translation of Rare Words with Subword Units
Radford et al. — Language Models are Unsupervised Multitask Learners (GPT-2)
openai/gpt-2 — source of the embedded merge table (Modified MIT License) · openai/tiktoken · OpenAI Tokenizer
Embedded data: the 10,497 most useful merges of GPT-2’s 50,000, selected by usage frequency over a reference corpus. On everyday English this page matches the full tokenizer for about 9 out of 10 words; rare words may split into smaller pieces here. Token IDs shown are the real GPT-2 IDs.