HOW
MACHINES
LEARN

An artificial neuron does three things: it takes some numbers in, multiplies each by a weight, adds a bias, and passes the total through an activation function to produce one number out. That's it. That is the atom of all deep learning.

Try it. This neuron decides whether to take an umbrella. Its inputs are how cloudy it is and how humid it feels. The weights are how much the neuron cares about each input; the bias is its baseline reluctance or eagerness before it has seen anything at all.

Notice what the activation function is for. Without it, a neuron is pure linear arithmetic — its output is always a flat, straight-line function of its inputs. The activation bends that line. This nonlinearity is the entire reason networks can learn curves, spirals and language rather than just straight lines. Flip between the options above and watch the curve change shape.

Arrange neurons in layers. The first layer receives the raw data — the input layer. The last produces the answer — the output layer. Everything in between is a hidden layer: "hidden" only in the sense that you never directly see its inputs or outputs. Each hidden neuron learns to detect some intermediate pattern, and later neurons combine those patterns into more abstract ones. When a network has many hidden layers we call it deep — that is the entire origin of the phrase deep learning.

Watch data flow through this network, left to right. That flow — multiply, add, activate, layer by layer — is called the forward pass. When you use a trained model (every time you ask a chatbot something), you are running forward passes. That's also called inference.

Every connection has its own weight; every neuron its own bias. Together these adjustable numbers are the network's parameters — its entire memory and personality. Count them for the network above:

Hold that number. The playground network below has around 50–200. GPT-3 2020 had 175,000,000,000. Frontier models are estimated around a trillion. Same atom, same arithmetic — just unimaginably more of it. The overall shape you choose — how many layers, how wide, how they connect — is called the architecture.

Step 1 — measure the error. Show the network an example where you know the right answer (that's training data), compare its output to the truth, and boil the difference down to a single number: the loss. High loss = badly wrong. Zero loss = perfect. Training has exactly one goal: make this number go down.

Step 2 — find which way is downhill. Imagine a landscape where every possible setting of the weights is a location, and the altitude at each location is the loss. Learning is finding a low point in that landscape. The gradient is the local slope — it tells you, for every single weight, which direction makes the loss worse. So you step the other way. Do this over and over and you are performing gradient descent, the algorithm at the heart of essentially all modern AI.

Step 3 — take a step of the right size. How big a step? That's the learning rate — the single most temperamental dial in the field. Try it yourself:

Step 4 — assign the blame. One puzzle remains, and it stalled the field for decades: in a deep network, how do you know how much each individual weight — including ones buried in hidden layers — contributed to the final error? The answer is backpropagation: after the forward pass, walk the error backwards through the network, layer by layer, using calculus's chain rule to split the blame at every connection. It's an accounting trick, really — a supremely efficient way to compute the gradient for every parameter at once. Popularized in 1986, it is still how every frontier model learns today.

Two last pieces of vocabulary and the loop is complete. In practice you don't compute the error on one example at a time but on a small batch of them (this page uses 16). One full pass through the entire training set is an epoch. Training a model means running this loop — forward pass, loss, backpropagation, weight update — for thousands or, at frontier scale, trillions of steps. Because each batch is a random sample, the method is properly called stochastic gradient descent.

Each dot is a training example with two input numbers (its x and y position) and a label (red or blue). The network's task is classification: predict the label from the position. The painted background shows its current opinion about every point in the plane — the decision boundary is where the colour flips. Hollow dots are test data: examples the network never trains on, kept aside to check it has genuinely generalized rather than memorized.

Experiments worth running

• Delete all hidden layers (− LAYER twice) on BLOBS: still solves it. One layer can draw one straight line, and a line is all blobs need. This is 1958 in miniature.
• Now switch to XOR with no hidden layers. It flails forever — no straight line separates a checkerboard. This precise failure, published in 1969, helped freeze neural-network funding for a decade.
• Add one hidden layer back. XOR falls in seconds. Hidden layers + backpropagation = the 1986 comeback.
• Set activation to LINEAR on the spiral, with 3 big hidden layers. Watch it fail no matter how deep: stacked linear layers collapse into one straight line. Depth is worthless without nonlinearity. TRY IT ↑
• Race TANH vs RELU on the spiral (2 layers of 8). ReLU usually trains visibly faster — a small taste of why it helped ignite the deep-learning boom in 2012.
• Crank the learning rate to 3.0. The loss curve goes feral. You've met this ball before, in section 03.

To a network, your drawing is just numbers: the 8×8 pad below becomes 64 inputs, each 0 (empty) or 1 (inked). Those 64 numbers are the features. A [64 → 12 → 1] network — 64 inputs, 12 hidden neurons, 1 output — will learn to map them to a class. That is, at miniature scale, exactly how early neural networks read handwritten ZIP codes for the US postal service in the 1990s.

How to play: draw a shape (say, an ✕) on the pad, add it to CLASS A. Draw an ○, add to CLASS B. Give each class at least 3–4 varied examples, hit TRAIN, then draw fresh shapes on the test pad and watch the network's live verdict.

What this teaches (and what breaks)

• With 3 examples per class, the network can ace its training set while understanding nothing. Perfect scores on data it has already seen prove little — that failure mode is overfitting, and it's why the test pad exists.
• Draw your test ✕ shifted one pixel to the corner. It will often fail — this network has no idea that a shape moved sideways is the same shape. Fixing exactly that weakness is why convolutional networks were invented in 1989: they scan for patterns everywhere in an image instead of memorizing pixel positions.
• More varied examples → better generalization. You have just discovered the most reliable law in machine learning: data quality and quantity usually beat cleverness.

What is identical. Frontier models are made of the same neurons computing weights-times-inputs-plus-bias. They learn by the same loop: forward pass, loss, backpropagation, gradient descent (with a smarter step-size scheme called Adam, from 2014). If you followed sections 01–04, you understand the core mechanism of a trillion-dollar industry. Genuinely.

What is different. Three things, mainly:

• Architecture. Modern language models are transformers 2017, whose key component — attention — lets every word in a sentence look directly at every other word and decide which ones matter for interpreting it — WATCH IT HAPPEN →. Words enter the network as embeddings: learned lists of numbers where similar meanings sit near each other, just as your drawings entered as 64 pixel numbers. How raw text becomes those numbers at all is its own story — SEE TEXT BECOME TOKENS →
• Scale. Researchers found that loss falls with eerie predictability as you grow the model, the data and the compute together — the scaling laws 2020. That discovery turned "build a bigger one" from a hunch into an investment thesis, and it is why training runs now consume city-scale quantities of GPU compute.
• Training recipe. A modern assistant is built in stages: pretraining (predict the next word across a huge slice of human text — gradient descent on trillions of examples), then fine-tuning on demonstrations, then RLHF — reinforcement learning from human feedback 2017–22 — where human preferences steer the model toward being helpful rather than merely predictive.

To feel the scale, count parameters — every bar below is on a logarithmic axis, where each gridline is 10× the last. On a linear chart, every bar except the final two would be invisibly thin.

Two more labs complete the language story

This page taught the universal machinery: neurons, layers, gradient descent. Language models add two mechanisms of their own — how text becomes numbers, and how words decide which other words matter. Each now has its own live laboratory, built with the same rules: real math, no libraries, view source.