Block 02 / 07
A model is a parameterized function fitted by optimization.
A model is a parameterized function. You show it inputs and desired outputs. A loss scores the miss. Gradient descent nudges the parameters so the miss shrinks. CNNConvolutional neural network. Learns local filters that slide across the input, exploiting spatial structure. The default shape for images before Vision Transformers arrived., RNNRecurrent neural network. Processes a sequence one step at a time, carrying a hidden state forward. Stepping through that state sequentially is hard to parallelize, part of why attention displaced it., TransformerThe architecture built on self-attention (block 03). Every token attends to every other token in the window, computed in parallel rather than stepped through sequentially like an RNN., state-spaceState-space model. Processes a sequence through a compressed, continuously updated state, built for the parallel training and long-context efficiency that made Transformers dominant. Mamba is a current example., mixture-of-expertsA layer made of several parallel sub-networks, the experts, plus a router that sends each token to only a few of them. Total parameter count grows while inference cost stays close to that of a much smaller model.: different shapes, same job. Inference is running the fitted function. Fine-tuning is a smaller write to the same parameters.
A new architecture is a new parameterization of the same problem. The relevant questions are inductive bias, training cost, and inference cost. Alignment and instruction tuning change the objective, not the kind of object being optimized.
Step 0 of 200 · barely trained, gradients just started
Drag the slider to scrub through training steps.
Three families of optimizer do the walking that turns a computed gradient into an updated parameter. Each answers the same question, how far to step and in what direction, with a different tradeoff among speed, memory, and stability.
Classic, full-batch
Computes the gradient over the entire training set before taking a single step, exact and stable. A full pass through the data for every step made it impractical at the scale modern models train on.
The practical default for decades
Steps on small batches instead of the full dataset, trading exactness for speed. Momentum keeps a running average of past gradients, smoothing the noise from any one batch and building speed in a direction that keeps proving useful.
The production default today
Adapts a separate step size per parameter from running estimates of the gradient’s mean and variance (Kingma and Ba, 2014). AdamW decouples weight decay from that adaptation. Little tuning is required to train a large TransformerThe architecture built on self-attention (block 03). Every token attends to every other token in the window, computed in parallel rather than stepped through sequentially like an RNN., which is why it trains most production models today.
Paper
A scalar loss, a gradient, and a parameter update. Subsequent architectures reuse this training procedure.
Book
Free online. The optimization and backpropagation chapters state what “the model learned” means in terms of a loss and a parameter update.
Course
The early lectures derive a network as a function, a loss, and SGD, with assignments you can run. Vision is the example. The math transfers.
A model is a fitted function. Inference is running that function. The following systems are that job under different inputs and losses.
An image, a click, or a query is mapped to a score. The parameters were fitted so that score matches a labeled outcome. A CNN, a linear ranker, and a Transformer classifier are different shapes of the same object.
A language model is a function from a token sequence to a distribution over the next token. Serving it is evaluating that function. Fine-tuning is a further write to the same parameters.
Tokens in, tokens out, latency, and dollars are properties of evaluating a fitted function at a given size. Architecture names matter only insofar as they change that bill.
In numpy (or a thin autograd), implement SGD on a tiny MLP that fits a known function (a 2D surface, a toy classifier). Then swap the MLP for a small attention block, or a tiny conv net, without changing the training loop: batch, loss, backward, step. Plot train and held-out loss for both shapes.