Neural Networks, Activations and Backpropagation

By the end of this lesson you will be able to read “thirty-six layers, hidden size four thousand, gated SiLU activations, residual connections” and know what each phrase is doing there; count the parameters of a layer from its shape; and follow a number computed at the very end of a network back to a weight at the very beginning, by hand, with the arithmetic checked by PyTorch. That last mechanism, backpropagation, is what every training run in this course has in common, from the digit classifier in this part’s lab to a fine-tune of an eight-billion-parameter model in Part 13.

The four scripts on this page need only the torch the lab installs, run on the CPU in seconds, and print the output shown beneath each (torch 2.14.0 on a CPU; seeded random values may differ in the last digits on another build).

A neuron is a weighted sum with a decision

A neuron takes some inputs, multiplies each by a weight, adds the products together, adds a bias, and passes the total through a function that decides what to output. For inputs x₁ … xₙ with weights w₁ … wₙ and bias b:

Pseudocode — not a real command

z = w1*x1 + w2*x2 + ... + wn*xn + b     # the pre-activation: a weighted sum
output = activation(z)                  # the nonlinearity

With three inputs, three weights and a bias, the whole computation is one row of a table:

Term	Value
Inputs x	[1.0, 2.0, -1.0]
Weights w	[0.5, -0.25, 1.0]
Bias b	0.1
z = 0.5×1.0 + (−0.25)×2.0 + 1.0×(−1.0) + 0.1	−0.9
relu(z) = max(0, z)	0.0

The weighted sum is the part that learns: the weights and the bias are the parameters that the previous lesson’s gradient descent adjusts. The activation is what makes the neuron more than a weighted sum. What matters about each is its formula and its derivative, because the derivative is what the backward pass multiplies by:

Activation	Formula	Derivative	Where you meet it
ReLU	max(0, z)	1 for z > 0, 0 otherwise (PyTorch takes 0 at exactly zero)	The lab’s digit classifier
Sigmoid σ	1 / (1 + e^(−z))	σ(z)(1 − σ(z)), at most 0.25	Gates; the output of a two-class classifier
tanh	(e^z − e^(−z)) / (e^z + e^(−z))	1 − tanh(z)², at most 1	Older networks; this page’s depth experiment
SiLU (swish)	z × σ(z)	σ(z)(1 + z(1 − σ(z)))	hidden_act: "silu" in Qwen3’s config.json
Gated SiLU (SwiGLU)	silu(W_gate x) × (W_up x), then W_down	Product rule over both branches	The feed-forward sub-layer of every Qwen3 block

Without an activation a neuron is linear, and a stack of linear layers is one linear layer however tall the stack. The script shows it with numbers: a layer of four neurons as one matrix multiplication, then two linear layers computing exactly what one layer with the product matrix computes, until a ReLU sits between them.

RunnableAll tracks

one-neuron.py

import torch

# One neuron: three inputs, three weights, one bias, one ReLU.
x = torch.tensor([1.0, 2.0, -1.0])
w = torch.tensor([0.5, -0.25, 1.0])
b = 0.1
z = (w * x).sum() + b                      # 0.5 - 0.5 - 1.0 + 0.1
print(f"z = {z.item():.2f}   relu(z) = {torch.relu(z).item():.2f}")

# A layer of four neurons reading the same three inputs is one matrix multiplication.
# PyTorch stores one row per neuron, so W has shape (4, 3): (outputs, inputs).
W = torch.tensor([[0.5, -0.25, 1.0],
                  [0.1,  0.1,  0.1],
                  [-1.0, 0.0,  0.5],
                  [0.0,  0.3, -0.3]])
bias = torch.tensor([0.1, 0.0, 0.2, -0.1])
z = W @ x + bias
print("layer z     :", [round(v, 2) for v in z.tolist()])
print("layer relu  :", [round(v, 2) for v in torch.relu(z).tolist()])

# Without a nonlinearity two layers are one layer: W2 @ (W1 @ x) == (W2 @ W1) @ x.
torch.manual_seed(0)
W1, W2 = torch.randn(4, 3), torch.randn(2, 4)
print("two linear layers    :", [round(v, 4) for v in (W2 @ (W1 @ x)).tolist()])
print("one layer, W2 @ W1   :", [round(v, 4) for v in ((W2 @ W1) @ x).tolist()])
print("relu between the two :", [round(v, 4) for v in (W2 @ torch.relu(W1 @ x)).tolist()])

Output — what you should see

one-neuron.py, torch 2.14.0, CPU

z = -0.90   relu(z) = 0.00
layer z     : [-0.9, 0.2, -1.3, 0.8]
layer relu  : [0.0, 0.2, 0.0, 0.8]
two linear layers    : [-6.1224, -1.6618]
one layer, W2 @ W1   : [-6.1224, -1.6618]
relu between the two : [-5.6818, -4.2706]

The first row of W is the neuron from the table. The last three lines are the whole argument for activation functions: two linear layers equal one matrix, so ten would too; the ReLU zeroed two of the four intermediate values and broke the equality.

✕A dead ReLU never comes back

ReLU’s derivative below zero is exactly zero, so a hidden unit whose weighted sum is negative for every input in the batch sends no gradient to its weights, and gradient descent cannot move it. The symptom is a fraction of hidden units that output zero for every input, and it grows after a step that was too large; zero-init.py below counts 17 of the lab network’s 256 hidden units silent on a random batch before training has begun. Fixes, in order: a smaller learning rate; an activation with a nonzero slope below zero: SiLU, which Qwen3’s config.json selects as hidden_act, has one.

A layer is many neurons: one matrix multiplication

Put out neurons side by side, each with its own row of in weights reading the same in inputs, and the layer is the matrix multiplication W @ x + b, with W of shape (out, in) and b of length out, then the activation on every element. That is the shape the lab prints for its first layer, layers.0.weight shape (256, 784), and why the next lesson says matrix multiplication is everything. The parameter count follows from the shape, in × out weights plus out biases, and for the lab’s network the four tensors it prints add up to the count it prints:

Tensor	Shape	Parameters
layers.0.weight	(256, 784)	200,704
layers.0.bias	(256,)	256
layers.2.weight	(10, 256)	2,560
layers.2.bias	(10,)	10
Total		203,530

The number of neurons in a layer is its width; the lab’s --hidden flag sets it. Both matrices in that network have the hidden size as one dimension and a fixed number as the other, so doubling the width doubles the parameters; in a language model’s block, where a matrix multiplies the hidden size by a multiple of itself, doubling the width quadruples them.

--hidden	Parameters	Compared with 256
128	101,770	half
256	203,530	the default
512	407,050	double
1024	814,090	four times

Widen when training loss and validation loss are both still high at the end of training, which the generalisation lesson calls underfitting; when validation loss rises while training loss keeps falling, width is not the fix, and the lab’s --hidden 512 row shows what each doubling costs in checkpoint bytes.

In a language model the widths are large. Qwen3-8B (Apache-2.0) states its shape in three fields of its config.json: hidden_size: 4096, intermediate_size: 12288, num_hidden_layers: 36. Its feed-forward sub-layer, in the transformers modelling code, is three bias-free matrices, a gate and an up projection from 4096 to 12288 and a down projection back, combined as down(silu(gate(x)) × up(x)). Arithmetic on those fields, not a measurement:

Piece of one block	Shape arithmetic	Parameters
Gate, up and down projections	3 × 4096 × 12288	150,994,944
The four attention matrices (Part 2 explains them)	4096 × 4096 × 2 + 4096 × 1024 × 2	41,943,040
Normalisation scales	2 × 4096 + 2 × 128	8,448
One block		192,946,432
36 blocks	36 × 192,946,432	6,946,071,552
Input embedding and output projection	2 × 151,936 × 4096	1,244,659,712
Whole model, with the final norm		8,190,735,360

Width is where the parameters live: the feed-forward matrices are 78 per cent of each block and the blocks 85 per cent of the model. The count lands on the model card’s 8.2 billion, and Part 2’s lab checks it against the tensors of a real checkpoint.

Depth: layers reading layers

Feed one layer’s outputs into another as its inputs, and again, and the network is deep. Each layer builds on what the previous one computed: in a digit classifier the first layer’s units respond to strokes, later ones to arrangements of strokes; the lab’s notebook cell that draws sixteen rows of layers.0.weight as 28×28 images shows those strokes. Nobody designs those features; they are whatever the weights settle into. A composition of simple functions represents things a single wide layer represents only clumsily. Depth is also what makes backpropagation a long chain, which is why the last two sections of this page exist.

Network	Depth	Where the number comes from
The lab’s digit classifier	2 linear layers	train-mnist.py
Qwen3-8B	36 blocks, each with an attention and a feed-forward sub-layer	num_hidden_layers in config.json
Qwen3-235B-A22B (Apache-2.0), the course’s largest reference model	94 blocks	the model’s configuration, as recorded in the course’s model data

Parameters versus hyperparameters

Two words that sound alike and mean opposite things. The test is one question: does training change it?

Parameters are the weights and biases: the 203,530 numbers that loss.backward() puts a gradient on and optimiser.step() moves; “an 8B model” counts them. Hyperparameters are the choices made about training and architecture. Training never touches them; you set them, and you judge them by the validation loss the generalisation lesson describes, never by the training loss.

Setting in train-mnist.py	Kind	What it decides
--lr 0.1	Hyperparameter	The step size of every update
--batch-size 128	Hyperparameter	How many examples one gradient is averaged over; also the unit of memory
--epochs 5	Hyperparameter	How many passes over the training set
--hidden 256	Hyperparameter (architecture)	The width of the hidden layer, and so the parameter count
--seed 0	Hyperparameter (bookkeeping)	Which random initial weights and which batch order
layers.0.weight, (256, 784)	Parameters	Learned; never typed by anyone

When Part 13 says “rank 16, alpha 32” about a LoRA fine-tune, those are hyperparameters of the adapter, and that lesson explains what raising the rank at a fixed alpha does to the adapter’s contribution, and why alpha = 2r is a common convention. The initialisation scale is a hyperparameter too, and it decides whether training starts at all.

Backpropagation: the chain rule, run backwards

Gradient descent needs, for every parameter, the gradient of the loss with respect to it: nudge this weight up a little, does the loss go up or down, and how steeply? The loss is computed at the end of the network, and a weight in the first layer influences it only through everything after it. Trying every nudge one at a time would cost one forward pass per parameter, billions per step.

The answer is the chain rule. If the loss depends on y, and y depends on h, and h depends on a weight w, then

Pseudocode — not a real command

dloss/dw = dloss/dy × dy/dh × dh/dw

Every link is a local derivative a layer can compute from its own inputs. Backpropagation is that product organised so that nothing is computed twice: start with dloss/dy at the output, multiply back through one layer to get the gradient with respect to its inputs, hand that to the layer before, and at every layer split off the gradient with respect to its own weights on the way past. Here is the whole computation for a network small enough to do on paper: two inputs, two ReLU hidden units, one output, squared-error loss, target 3.0.

Forward	Value	Backward	Value
z_A = 0.5×1 + 0.5×2 + 0	1.5	dloss/dy = y − t	−1.0
z_B = −1.0×1 + 0.25×2 + 0	−0.5	dloss/dv = dloss/dy × h	[−1.5, 0.0]
h = relu(z)	[1.5, 0.0]	dloss/dc = dloss/dy	−1.0
y = 1.0×1.5 + 2.0×0.0 + 0.5	2.0	dloss/dh = dloss/dy × v	[−1.0, −2.0]
loss = ½ (y − 3.0)²	0.5	dloss/dz = dloss/dh × relu'(z)	[−1.0, 0.0]
		dloss/dW1 = dloss/dz ⊗ x, dloss/db1 = dloss/dz	[[−1.0, −2.0], [0.0, 0.0]], [−1.0, 0.0]

Unit B’s weighted sum was negative, so its ReLU derivative is zero and so is every gradient behind it: this example teaches unit B nothing, the dead-ReLU mechanism seen from the backward side. The script builds the same network and lets torch.autograd check the table, then takes one gradient-descent step and runs the forward pass again.

RunnableAll tracks

backprop-autograd.py

# The paper network from the table, built in PyTorch so autograd can check the arithmetic,
# then one gradient-descent step and the forward pass again.
import torch

x = torch.tensor([1.0, 2.0])                                        # the input
t = 3.0                                                             # the target
W1 = torch.tensor([[0.5, 0.5], [-1.0, 0.25]], requires_grad=True)  # hidden units A and B
b1 = torch.zeros(2, requires_grad=True)
v = torch.tensor([1.0, 2.0], requires_grad=True)                    # output weights
c = torch.tensor(0.5, requires_grad=True)                           # output bias

def forward():
    h = torch.relu(W1 @ x + b1)
    return h, v @ h + c

h, y = forward()
loss = 0.5 * (y - t) ** 2
print(f"h = {h.tolist()}  y = {y.item()}  loss = {loss.item()}")
loss.backward()                                                     # autograd runs the chain rule
print(f"dloss/dv = {v.grad.tolist()}  dloss/dc = {c.grad.item()}")
print(f"dloss/dW1 = {W1.grad.tolist()}  dloss/db1 = {b1.grad.tolist()}")

lr = 0.1
with torch.no_grad():                                               # the update is not recorded
    for p in (W1, b1, v, c):
        p -= lr * p.grad                                            # w = w - lr * grad, as SGD does
h, y = forward()
print(f"after one step: y = {y.item():.4f}  loss = {0.5 * (y.item() - t) ** 2:.6f}")

Output — what you should see

backprop-autograd.py, torch 2.14.0, CPU

h = [1.5, 0.0]  y = 2.0  loss = 0.5
dloss/dv = [-1.5, -0.0]  dloss/dc = -1.0
dloss/dW1 = [[-1.0, -2.0], [0.0, 0.0]]  dloss/db1 = [-1.0, 0.0]
after one step: y = 3.0150  loss = 0.000112

Autograd’s gradients are the table’s to the digit. One step at learning rate 0.1 moved five parameters at once and took the output from 2.0 to 3.015, slightly past the target: a learning rate a little large for one example, and the previous lesson’s bouncing loss is this overshoot repeated.

One training step, with the backward pass shown

1. 1ForwardInputs flow through every layer to a prediction, then to a loss.
2. 2BackwardThe chain rule carries the loss gradient back through each layer to every weight.
3. 3UpdateEach weight moves a small step against its gradient.

The forward pass computes the loss; the backward pass distributes its gradient to every weight; the update moves each weight against its gradient. The previous lesson's loop repeats this over batches and epochs.

What a backward pass costs

A linear layer’s forward pass is one matrix multiplication. Its backward pass is two of the same size: the incoming gradient times the transposed weights, for the layer before, and the transposed inputs times the incoming gradient, for the weights. So a backward pass costs about twice a forward pass:

Pass	Matrix multiplications per linear layer	Operations per parameter per token
Forward	1 (x @ Wᵀ)	2 (a multiply and an add)
Backward	2 (δ @ W for the previous layer, xᵀ @ δ for the weights)	4
One training step	3	6

The last column is the rule of thumb the next lesson quotes and Part 12 uses to budget a pretraining run.

What the frameworks do

You will not write backpropagation by hand again in this course. PyTorch’s autograd records every operation on a tensor whose requires_grad is True; loss.backward() walks the record in reverse and fills in .grad on each such tensor. Two facts from the tutorial matter in practice: PyTorch accumulates gradients, adding each backward pass to what .grad already holds, which is why the lab’s loop calls optimiser.zero_grad() before every loss.backward(); and torch.no_grad() switches the recording off, which is why the lab’s evaluate costs a forward pass and no backward memory. MLX describes its automatic differentiation as working “on functions rather than on implicit graphs”: grad and value_and_grad turn a function into one that returns its gradient, which is what the lab’s Mac step uses.

▤Memory budget

The backward pass is why training needs so much more memory than inference. xᵀ @ δ needs x, the layer’s input from the forward pass, so every layer’s input is kept until the backward pass reaches it: the whole forward pass, for every token in the batch, held at once. Part 11’s memory lesson turns that into gigabytes for a named model and batch; it is the term activation checkpointing trades for recomputation.

Initialisation: where the weights start

Before the first step every weight needs a value, and two things can go wrong: the weights can be too alike, or the wrong size.

Too alike. Set every weight in a layer to the same value and every neuron computes the same weighted sum, receives the same gradient and takes the same step, forever: the layer is one neuron wide however many it has. Zero is worse: a hidden layer’s output is zero, so the next layer’s weight gradient xᵀ @ δ is zero, and the gradient passed back through zero weights is zero too. The script builds the lab’s network three ways and inspects the first layer’s weight gradient after one backward pass over a random batch:

RunnableAll tracks

zero-init.py

import torch
from torch import nn

torch.manual_seed(0)
images = torch.rand(128, 784)                 # a batch shaped like the lab's, random pixels
labels = torch.randint(0, 10, (128,))         # random digits
loss_fn = nn.CrossEntropyLoss()

def one_backward(init_value):
    net = nn.Sequential(nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
    if init_value is not None:
        for p in net.parameters():
            nn.init.constant_(p, init_value)
    loss = loss_fn(net(images), labels)
    loss.backward()
    W1 = net[0].weight.grad                   # shape (256, 784): one row per hidden unit
    label = "PyTorch default" if init_value is None else f"all weights {init_value}"
    print(f"{label:16s} loss {loss.item():.4f}   "
          f"distinct rows in W1.grad: {torch.unique(W1, dim=0).shape[0]:3d}/256   "
          f"rows that are all zero: {int((W1 == 0).all(dim=1).sum()):3d}/256")

one_backward(0.0)
one_backward(0.01)
one_backward(None)

Output — what you should see

zero-init.py, torch 2.14.0, CPU

all weights 0.0  loss 2.3026   distinct rows in W1.grad:   1/256   rows that are all zero: 256/256
all weights 0.01 loss 2.3026   distinct rows in W1.grad:   1/256   rows that are all zero:   0/256
PyTorch default  loss 2.3084   distinct rows in W1.grad: 240/256   rows that are all zero:  17/256

With zeros, nothing in the first layer gets a gradient. With a constant, every hidden unit gets the same gradient row and after the step they are still identical: a 256-wide layer one unit wide. With the default the rows differ, except the 17 units silent on this batch, the dead ReLUs from the first section. All three start at or near ln(10) = 2.3026, the cost of spreading probability evenly over ten digits, which is what any correctly initialised classifier prints before training.

The wrong size. The weights must be random at the right scale, and the scale depends on how many inputs the layer has, because a weighted sum of in random terms grows with in. PyTorch’s nn.Linear documentation states its default: weights and biases drawn from U(−√k, √k) with k = 1/in_features. The named alternatives in torch.nn.init come from the papers that worked out why the scale matters: xavier_uniform_ (Glorot and Bengio, 2010) uses √(6 / (fan_in + fan_out)) and kaiming_uniform_ (He et al., 2015, derived for ReLU) uses gain × √(3 / fan_in). A transformer’s config.json carries the choice as one number: Qwen3-8B’s initializer_range: 0.02 is the standard deviation of the normal distribution the transformers code draws a freshly built model’s weight matrices from.

Layer	Rule	Bound or standard deviation
nn.Linear(784, 256), the lab’s first layer	PyTorch default, √(1/784)	±0.0357
nn.Linear(256, 10), the lab’s second layer	PyTorch default, √(1/256)	±0.0625
nn.Linear(784, 256)	kaiming_uniform_, gain √2 for ReLU	±0.0875
Any weight matrix in Qwen3-8B, when built from scratch	normal, initializer_range	std 0.02

Why the scale decides whether training starts is the next section’s product of many factors. What you need in practice is the table of symptoms:

Symptom at the first steps	Cause	Check or change
Loss starts near ln(classes) and does not move	Learning rate too small, or no gradient reaching the weights: constant initialisation, dead units, a vanished product	The lab’s --lr 0.0001 run: 2.3053 after a full epoch. Multiply the learning rate by 10; print sum(p.grad.norm() ** 2 for p in model.parameters()) ** 0.5 after loss.backward(); a value near zero confirms no gradient is reaching the weights
Loss starts far above ln(classes)	Initial logits too large: weights too big for the fan-in, or inputs not scaled to a small range	Scale the inputs (the lab’s ToTensor() maps pixels to 0–1); use the default initialiser
Loss becomes nan or inf within a few steps, or jumps to hundreds	Learning rate too large, or an exploding product through depth	The lab’s --lr 20 run: 395.57 in epoch 1. Divide the learning rate by 10; clip with torch.nn.utils.clip_grad_norm_
Loss exactly ln(classes) and every hidden gradient zero	Zero initialisation	Use the framework’s default

ln(classes) for the lab is 2.3026; its epoch-0 line prints 2.3091. For a language model the classes are the vocabulary: Qwen3-8B’s vocab_size of 151,936 makes the even guess cost ln(151936) = 11.93, and the first loss: line the Part 12 lab prints should sit near the logarithm of its tokeniser’s vocabulary for the same reason.

Why depth needs residual connections

The backward pass through L layers multiplies L local derivatives together, and a product of many factors is tiny or enormous unless every factor is close to one:

Factor per layer	20 layers	50 layers	94 layers
0.9	0.12	0.005	0.00005
1.1	6.7	117	7,800

A factor of 0.9 is what a tanh unit slightly off centre gives; a sigmoid gives at most 0.25. Glorot and Bengio traced the difficulty of deep networks to exactly this, the per-layer scaling drifting from one, and initialisation rules try to start every factor near one. They can only arrange that at step zero; training moves the weights.

The fix that made modern depth possible is the residual connection. He et al. (2015) describe it as reformulating “the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions”: instead of y = f(x), a layer computes

Pseudocode — not a real command

y = x + f(x)          # the layer's job is the change, not the whole
dy/dx = 1 + f'(x)     # the pass-through contributes exactly one

Whatever f' does, the gradient can always take the 1. Across L layers the factors are (1 + f'ᵢ), all near one when f' is small, so the loss gradient reaches the first layer at a usable size however many layers are stacked; their paper trained 152. The script measures it: fifty tanh layers of width 256 at three weight scales, as a plain stack and then as residual blocks, reporting the activations after the last layer and the gradient on the first layer’s weights against the last.

RunnableAll tracks

depth-and-residuals.py

import torch

torch.manual_seed(0)
depth, width, batch = 50, 256, 64
x = torch.randn(batch, width)

def run(scale, residual):
    """Push a batch through `depth` tanh layers and report whether the gradient reaches layer 1."""
    Ws = [(torch.randn(width, width) * scale / width ** 0.5).requires_grad_() for _ in range(depth)]
    h = x
    for W in Ws:
        if residual:                      # pre-norm residual block: h + f(norm(h)), as a transformer does
            inp = h / torch.sqrt((h * h).mean(dim=1, keepdim=True) + 1e-6)
            h = h + torch.tanh(inp @ W)
        else:                             # plain stack: each layer replaces its input
            h = torch.tanh(h @ W)
    h.sum().backward()
    kind = "residual+norm" if residual else "plain"
    print(f"{kind:13s} std {scale:.1f}/sqrt(256)   |h| at layer 50: {h.std().item():8.2e}   "
          f"grad norm, layer 1: {Ws[0].grad.norm().item():8.2e}   layer 50: {Ws[-1].grad.norm().item():8.2e}")

for scale in (0.5, 1.0, 2.0):
    run(scale, residual=False)
for scale in (0.5, 1.0, 2.0):
    run(scale, residual=True)

Output — what you should see

depth-and-residuals.py, torch 2.14.0, CPU

plain         std 0.5/sqrt(256)   |h| at layer 50: 5.59e-16   grad norm, layer 1: 2.43e-12   layer 50: 2.22e-12
plain         std 1.0/sqrt(256)   |h| at layer 50: 1.08e-01   grad norm, layer 1: 1.46e+02   layer 50: 2.35e+02
plain         std 2.0/sqrt(256)   |h| at layer 50: 7.23e-01   grad norm, layer 1: 2.11e+06   layer 50: 7.86e+02
residual+norm std 0.5/sqrt(256)   |h| at layer 50: 3.10e+00   grad norm, layer 1: 4.37e+03   layer 50: 1.63e+03
residual+norm std 1.0/sqrt(256)   |h| at layer 50: 4.54e+00   grad norm, layer 1: 6.93e+03   layer 50: 1.43e+03
residual+norm std 2.0/sqrt(256)   |h| at layer 50: 5.75e+00   grad norm, layer 1: 1.05e+04   layer 50: 9.89e+02

Read the plain rows against the table of products. At half the right scale the signal has shrunk to 1e-16 by the last layer and every gradient is 1e-12: nothing to learn from, the first row of the symptom table. At twice the right scale the tanh units saturate and the first layer’s gradient is 2,700 times the last’s: the first step wrecks the early layers and the loss soon reads nan, the third row. Only the middle row is usable, and a plain stack depends on staying there. The residual rows are the point: with the same three initialisations, good and bad, the first layer’s gradient stays within about a factor of ten of the last’s (2.7, 4.8 and 10.6 times, against 2,700 for the plain stack). The stack has become forgiving of its initialisation, and that is what lets thirty-six or ninety-four blocks train.

The residual rows also show why residual connections travel with normalisation: adding a bounded increment fifty times makes the stream grow, from a standard deviation of one at the input to between three and six at the output, and without the per-layer rescaling in the script that growth would feed every inp @ W. RMS normalisation, which every Qwen3 block applies before each sub-layer, divides a vector by its root mean square and multiplies by a learned scale:

Pseudocode — not a real command

rmsnorm(x) = x / sqrt(mean(x²) + eps) × g     # eps is rms_norm_eps in config.json, 1e-06 for Qwen3-8B

The 1e-6 in the script is that eps, there to stop a division by zero; the transformers implementation computes the mean of squares in FP32 whatever the model’s precision. Every transformer block in this course is built this way, and when Part 2’s lesson on the transformer talks about the “residual stream” running through a language model, it means the x in y = x + f(x): a channel each block reads from and adds to, so that depth is a sequence of small edits rather than a chain of transformations.

Check a gradient without trusting the training framework

For a scalar example, let the prediction be w × x, with squared loss (w × x − y)². Choose x = 2, y = 3 and w = 1. The prediction is two, the loss is one, and the derivative with respect to the weight is 2 × (2w − 3) × 2, which equals minus four. A small positive change in the weight should therefore reduce the loss.

Now approximate the derivative by evaluating the loss at w + ε and w − ε, subtracting, and dividing by 2ε. This finite-difference calculation should approach minus four for a suitable small epsilon. Extremely tiny epsilon can amplify floating-point cancellation, so numerical checks also have a precision budget.

This is a debugging technique, not how a large model should be trained: perturbing every parameter separately is expensive. Backpropagation reuses intermediate derivatives. If a custom operation trains badly, check a tiny instance numerically first. Also distinguish a correct gradient from a useful learning rate: even the correct downhill direction can overshoot when the step is too large.

Recap

A neuron is a weighted sum followed by a nonlinearity, and without the nonlinearity a stack of layers is one matrix. A layer is in × out + out parameters, and in a language model most of them sit in the wide feed-forward matrices of each block. Parameters are what training changes; hyperparameters are what you change. Backpropagation is the chain rule organised so that one backward pass, two matrix multiplications per layer against the forward pass’s one, gives every parameter its gradient, at the price of keeping every layer’s input in memory until it is used. Weights start random at a scale set by the fan-in, because identical weights stay identical and the wrong scale makes a product of fifty derivatives vanish or explode. Residual connections put a 1 into every factor of that product, which is why the models you will run can be as deep as they are.

Check your understanding

Question 1. A network of ten layers has no activation functions between them. What can it represent?

• Any function, because it is deep
• Only linear functions, because a composition of linear functions is linear
• Only functions of one variable
• Nothing until it is trained

Show the answer and why

Answer: Only linear functions, because a composition of linear functions is linear

The product of ten matrices is one matrix: one-neuron.py prints identical outputs for W2 @ (W1 @ x) and (W2 @ W1) @ x. Ten linear layers have more parameters than one, not more functions.

Question 2. Which of these are hyperparameters rather than parameters? Select all that apply.

• The learning rate
• A weight in the third layer
• The number of layers
• The initialisation scale
• A bias in the output layer

Show the answer and why

Answer: The learning rate, The number of layers, The initialisation scale

Weights and biases are parameters: training changes them. The learning rate, the depth and the initialisation scale are choices training never touches, each judged by the validation loss it produces.

Question 3. A plain fifty-layer stack, no residual connections, scales the gradient by about 0.8 at every layer. Roughly how large is the gradient reaching the first layer, relative to the gradient at the last?

• About 0.8 of it
• About 0.8 × 50 = 40 times smaller
• About 0.8^50, on the order of one hundred-thousandth
• The same, because gradients do not change size between layers

Show the answer and why

Answer: About 0.8^50, on the order of one hundred-thousandth

The factors multiply, so fifty of them compound to 0.8^50, about 1.4 × 10^-5. With residual connections each factor is 1 + f', near one when f' is small, and the first layer receives a gradient of the same order as the last.

Question 4. Which training loop is the bug?

• loss = loss_fn(model(x), y); optimiser.zero_grad(); loss.backward(); optimiser.step()
• optimiser.zero_grad(); loss = loss_fn(model(x), y); loss.backward(); optimiser.step()
• loss = loss_fn(model(x), y); loss.backward(); optimiser.step()
• optimiser.zero_grad(); loss = loss_fn(model(x), y); loss.backward(); optimiser.step(); del loss

Show the answer and why

Answer: loss = loss_fn(model(x), y); loss.backward(); optimiser.step()

The autograd tutorial states that PyTorch accumulates gradients: each backward pass adds to .grad. Without zero_grad() the third loop steps along the sum of every gradient so far, and the loss climbs. Where zero_grad() sits relative to the forward pass does not matter; its absence does.

Question 5. A freshly initialised classifier over 1,000 classes prints a loss of 6.9 on its first batch. Another prints 25. What do the two numbers say?

• Both are broken; a fresh network should print a loss near zero
• 6.9 is what an even guess over 1,000 classes costs, ln(1000) = 6.908, so that initialisation is healthy; 25 means the initial logits are already large and confidently wrong
• 25 is better, because a larger loss gives a larger gradient
• Neither number means anything until an epoch has run

Show the answer and why

Answer: 6.9 is what an even guess over 1,000 classes costs, ln(1000) = 6.908, so that initialisation is healthy; 25 means the initial logits are already large and confidently wrong

Cross-entropy charges −ln(probability of the correct class), and small initial weights spread probability evenly, costing ln(classes). A first loss well above that says the output layer already produces large logits; check the initialisation scale and the input scaling before touching the learning rate.

Check my answersClear