Tokenizer Studio — Peek LLM

Token stream

The assembly line is idle.
Pick a token above to watch how BPE built it, merge by merge.

Inspector

0tokens

0characters

context window0 / 4096

Tip. BPE is greedy for frequency: the pairs it merged earliest (lowest rank) are the pieces you will see everywhere — common words end up as single tokens, rare words shatter into familiar parts.

Sources. Sennrich et al. — BPE for NMT · Radford et al. — GPT-2 · openai/gpt-2 (Modified MIT License) · tiktoken
Embedded data: the 10,497 most useful GPT-2 merges by reference-corpus usage, documented next to the vendor block at the end of this file. Matches the full tokenizer on about 9 out of 10 everyday English words.