Attention is the idea that made modern AI possible: every word gets to look back at every earlier word and decide which ones matter. It's usually invisible, buried under billions of parameters. This page puts a real trained transformer — small enough to see through — under glass. Type a sentence; every line you see is a genuine attention weight from a genuine model.
Older networks read a sentence like water through a pipe — each word squeezed through in order, distant words fading. The transformer's 2017 move: let every word query every other word directly.
For each token, the model computes three small vectors, and the names are wonderfully literal:
Each token's query is compared against every earlier token's key — a dot product, i.e. a match score. The scores go through a , which turns them into positive weights summing exactly to 1: an attention budget. The token then takes a weighted blend of everyone's values. High weight = "this word matters for understanding me". That blend flows forward, and prediction is built on it.
Two more pieces and you can read everything below. First, this model predicts the next word, so a token may only look backwards — the blanks out the future (that's why the map in the microscope is a triangle). Second, the model runs several attentions in parallel — — each free to learn a different notion of "what matters": one head might track grammar while another tracks names. This page's model has 8 heads across 2 layers, and you can inspect each one.
Build a sentence from the model's 67-word vocabulary (it's a small creature — see the honesty box). Click any token to see where its attention goes. The star demo is preloaded: to guess the adjective after "…because it was", the model must look back at both nouns.
The head labels in the microscope weren't designed in — they were discovered, by measuring each head's behaviour after training. That discipline has a name, and it's one of the liveliest fields in AI.
Training never tells a head what to do. It only says: predict the next word, and here's your share of the gradient. Yet out of that pressure, specialists condense. In this page's model, measurement found antecedent-lookup heads (they connect "it…was" back to the nouns), a name/induction head (it finds earlier repetitions and copies them forward), and a subject-lookup component that solves was/were by attending past distractor nouns to the true subject.
The astonishing part is that the same story repeats at a billion times the scale. When researchers opened up real LLMs, they found induction heads doing precisely what the one on this page does — spot a repeated pattern, attend back to what followed it last time, copy — and there is strong evidence these circuits are a large part of how big models learn from examples in their prompt (so-called in-context learning) 2022. Reverse-engineering trained networks into circuits like these is called . On a 60,000-parameter model you can do it with a for-loop. On a trillion-parameter model it is a research frontier — and one of the main hopes for making AI systems auditable rather than oracular.
Attention weights are the rare part of a neural network that arrives pre-drawn as a diagram: a number between every pair of words, begging to be rendered as lines. That's what makes this microscope possible — and it's why attention maps became the field's favourite window into what these systems are doing, even though they are only part of the story (the values, and everything downstream, matter too).
Everything you just saw is real, and it's important to be precise about what "real" means here:
Companion pages: How Machines Read (how text becomes tokens before any of this happens) and How Machines Learn (the neurons, gradients and backpropagation underneath it all).