Before an AI model can predict a single word, your text must become numbers. The chopping-up step is called tokenization — and it explains token pricing, context windows, and why models famously miscount the r's in "strawberry". On this page a real tokenizer learns its vocabulary in front of you, then reads your own words.
A neural network is arithmetic — weights times inputs. So the very first step of any language model is brutal: convert text into a sequence of numbers.
The unit of conversion is the . A tokenizer owns a fixed — a numbered list of text chunks — and its whole job is to rewrite your sentence as a list of IDs from that list. The model never sees your letters again; it sees ID 4013 ID 88 ID 1729…
So what should the chunks be? Whole words fail (there are infinitely many: names, typos, "unbelievableness"). Single characters work but make every sentence painfully long. The answer used by essentially every modern model is a 1994 compression trick adapted for translation in 2016: . It learns the chunks from data, and the algorithm is almost embarrassingly simple:
Frequent words end up as single tokens. Rare words shatter into pieces. Nothing is ever unrepresentable — worst case, a word falls all the way back to characters. Frontier models run exactly this recipe with vocabularies of 100,000–200,000 chunks learned from terabytes of text. The tokenizer below runs it on a 3-kilobyte corpus, live, so you can watch every step.
This is the training corpus — the only text the tokenizer will ever learn from. Each merge welds the most frequent adjacent pair into a new vocabulary entry. Hit MERGE ONCE and watch English structure assemble itself, greediest pair first.
The first merges are always the gluey bits of English: t+h, e+·, i+n. Then suffixes crystallize (ing·, er·, s·), then whole common words fuse into single tokens (the·, and·). Watch the "corpus in tokens" number fall with every merge — a tokenizer is, at heart, a compressor. That is not a metaphor: the algorithm was invented for file compression in 1994, and researchers increasingly think compression and understanding are two faces of the same coin.
Type anything. The tokenizer applies its learned merges — in the order it learned them — and shows exactly what a model would receive. Every coloured chip is one token: one ID, one "word" in the machine's private language.
Some experiments worth a minute: common vs rare — "the" is one token, but a rare word shatters into fragments, because the tokenizer never saw it enough to build it. Typos — "netwrok" costs more tokens than "network"; a typo literally makes text more expensive and harder for the model to read. Word families — notice how "network", "networks" and "networking" share chunks: the model gets morphology half for free.
Once you know models see token chunks — never letters, never quite words — a whole family of AI folklore stops being mysterious.
Real tokenizers are this exact algorithm with three upgrades: they start from bytes rather than characters (so emoji, Chinese, code — anything — is representable and nothing is ever "unknown"); they use pre-tokenization rules to decide where merges may cross spaces and punctuation; and they add special tokens marking things like the start of a conversation turn. GPT-2 2019 shipped byte-level BPE with 50,257 tokens; current frontier vocabularies run 100K–200K. The learning algorithm you just watched is the same one.
Tokens are how text gets into the machine. What the machine does with them — every token looking at every other token to work out what matters — is the subject of the companion page: The Attention Microscope. And the mathematics underneath it all lives at How Machines Learn.