Tokenization
Before a language model reads anything, text is cut into tokens and turned into integers. Pick a way to see it happen.
-
Tokenization Slides
Six screens, one idea each: watch text shatter into tokens, slice words against your intuition, and train a BPE merge table by hand.
6 slides · no scrolling→ -
Tokenizer Studio
A single-screen workbench — controls, token stage, inspector. Click any token to watch how BPE assembled it, merge by merge.
Single screen · deepest view→ -
Tokenizer Playground
Open-ended exploration of the same live tokenizer: type anything, play the slicing game, and replay BPE training at your own pace.
Scroll at your own pace→