Watch GPT-2 read.
The real GPT-2 tokenizer is embedded in this page. Ahead: type and watch text shatter into tokens, slice words against your own intuition, and train a BPE merge table by hand.
The sentence above is being cut into tokens — each with its real GPT-2 ID.
Your turn: type anything.
Every keystroke is tokenized live. The window bar shows what these tokens cost against a 4096-token context budget.
Edit the text — tokens update as you type.
Slice it like GPT-2.
Click between characters to place your cuts, then check how the tokenizer really splits it. Five rounds.
Train the vocabulary yourself.
BPE starts from single characters and merges the most frequent adjacent pair, over and over. It trains itself when you arrive.
Press “Merge one pair”, or watch it train itself.
Vocabulary · 0 merges
Tokens ≠ words ≠ characters.
Three live examples, computed by the embedded tokenizer right now.
Common words stay whole
Frequent words cost one token each — cheap.
Rare words shatter
Long words break into pieces the model has seen before.
其他文字按字节计价
Outside common English, GPT-2 falls back to raw bytes (dashed, shown as hex) — one Chinese character can cost three tokens.
Why any of this matters.
Attention, embeddings, KV cache — everything downstream operates on these token IDs, never on your text.
Context windows, latency, and API cost are all measured in tokens — not words, not characters.
Sources
Sennrich, Haddow & Birch — Neural Machine Translation of Rare Words with Subword Units
Radford et al. — Language Models are Unsupervised Multitask Learners (GPT-2)
openai/gpt-2 — source of the embedded merge table (Modified MIT License) · openai/tiktoken · OpenAI Tokenizer
Embedded data: the 10,497 most useful merges of GPT-2’s 50,000, selected by usage frequency over a reference corpus. On everyday English this page matches the full tokenizer for about 9 out of 10 words; rare words may split into smaller pieces here. Token IDs shown are the real GPT-2 IDs. Sibling exhibit: Tokenizer Playground (dark, free-scroll edition).