δ delta the backward pass, one operation at a time

The backward pass, one operation at a time.

Most explanations of backpropagation walk you carefully through the forward pass and then wave at the rest. Weights change. The loss goes down. Learning happens. The part that actually computes the gradient shows up as arrows pointing left.

This page shows that part. Three small networks run one operation at a time, and every number stays on screen while they do it.

How to read it

Every view works the same way. Step with the buttons or the ← → keys, and sit on a step as long as you like. Hover any symbol in the equation to light up the parts of the diagram it refers to, and hover the diagram to go the other way.

What is under it

No machine-learning library. Every forward and backward pass here is written out by hand, which is the whole point of the project. All of it is checked against finite differences on every render, and you can watch that check run.

Why δ

Backprop never computes $\partial L/\partial w$ directly. It computes $\delta$, the error signal at each layer, and reads the weight gradients off it. That quantity is the thing worth watching, so the tool is named after it.

A network runs one operation at a time here, small enough that every number fits on screen. The error appears at the output, a transposed multiply carries it back, an elementwise product gates it by the local slope, and an outer product turns a δ into a weight gradient.

Fourteen steps in total. Every value is shown to three significant figures, and nothing is hidden behind a hover.

Step with the buttons or the ← → keys. Hover a symbol to light up what it points at, or hover the diagram to go the other way. Click a weight to take its gradient apart.

Why δ is the right thing to compute

A network with $n$ parameters has $n$ partial derivatives to find. Computing them one at a time would mean $n$ passes through the network. Backprop gets all of them in a single pass by never computing $\partial L/\partial w$ directly. It computes something else first:

$$\delta^{(l)}_i \;=\; \frac{\partial L}{\partial z^{(l)}_i}$$

That is how much the loss cares about the pre-activation of unit $i$ in layer $l$. There are far fewer of these than there are weights, one per unit rather than one per connection. Once you have them, every weight gradient in the layer is a single multiplication away. That is the entire efficiency argument.

Route, then gate

Equation (5) is two operations stapled together, and they do different jobs. Pulling them apart is the difference between memorising the formula and understanding it, which is why each one gets its own step here.

Route. $\left(W^{(l+1)}\right)^{\top}\delta^{(l+1)}$ sends the error backward along the connections. Column $j$ of $W^{(l+1)}$ lists every way unit $j$ influenced the layer above. Read that column back and you get unit $j$'s share of the blame. Same numbers, read against the grain.

Gate. $\odot\,\sigma'\!\left(z^{(l)}\right)$ scales that blame by how responsive the unit actually was. A unit sitting deep in tanh's flat region barely moved when its input moved, so it cannot be blamed for much, and it passes almost nothing further back.

Route, gate, repeat. The rest is bookkeeping.

The derivation everyone skips

Equation (4) says $\delta^{(L)} = \hat{y} - y$, and most explanations simply assert it. It deserves better, because something surprising happens. The derivative of the softmax is a full $K \times K$ matrix. The derivative of cross-entropy is a vector of reciprocals. Their product is neither. It is the prediction minus the truth.

Start with the softmax Jacobian. With $\hat{y}_k = e^{z_k}\big/\sum_m e^{z_m}$,

$$\frac{\partial \hat{y}_k}{\partial z_i} \;=\; \hat{y}_k\left(\mathbb{1}[k = i] - \hat{y}_i\right)$$

The diagonal case follows from the quotient rule, the off-diagonal case from the fact that $z_i$ sits in every denominator. Cross-entropy contributes $\partial L/\partial \hat{y}_k = -y_k/\hat{y}_k$. Chain them:

$$\frac{\partial L}{\partial z_i} = \sum_k \frac{\partial L}{\partial \hat{y}_k}\frac{\partial \hat{y}_k}{\partial z_i} = -\sum_k \frac{y_k}{\hat{y}_k}\,\hat{y}_k\left(\mathbb{1}[k=i] - \hat{y}_i\right)$$

The $\hat{y}_k$ cancels. That is the step that makes the whole thing work, and it only happens because cross-entropy's reciprocal meets softmax's own output. What is left is elementary:

$$-\sum_k y_k\left(\mathbb{1}[k=i] - \hat{y}_i\right) = -y_i + \hat{y}_i\underbrace{\sum_k y_k}_{=\,1} = \hat{y}_i - y_i$$

Two things follow. Softmax and cross-entropy are a matched pair: use softmax with squared error instead and none of this cancellation happens, the $\hat{y}_k$ survives in a denominator, and gradients die exactly when the network is confidently wrong. And this is why real implementations fuse the two into one softmax_cross_entropy op. Not for speed, but because computing them separately means building a Jacobian that is guaranteed to collapse.

Why check at all

Every line of the backward pass on this page was derived by hand, and hand-derived gradients are wrong constantly. A transpose in the wrong place, a sum over the wrong index, a missing $\sigma'$. The symptom is not a crash. It is a network that trains a little worse than it should, which looks exactly like a network that wants a better learning rate. A gradient that is 10% wrong still points roughly downhill.

Reading the table

Nudge one parameter by $\pm h$, measure the loss twice, and compare $\bigl(f(w+h) - f(w-h)\bigr)/2h$ against what the backward pass claimed. Central differences have error $O(h^2)$ rather than $O(h)$, which is why $h = 10^{-5}$ buys around ten correct digits instead of five. Below $10^{-7}$ the derivation is right. Above $10^{-3}$ it is not. Change $h$ above and the error gets worse in both directions: too large and you are measuring curvature, too small and you are measuring round-off.

The same engine with the dense layers swapped for convolutions. An 8×8 image goes in. Three 3×3 kernels slide over it, a ReLU clips the negatives, a 2×2 max-pool halves it, a second convolution mixes the three channels, and twelve surviving numbers meet a dense layer and a softmax. 95 parameters, all of them on screen.

The forward half is the tour every CNN explainer gives, and it is worth taking. Click a cell in any feature map to see the arithmetic that produced it. Hover a kernel weight to light up every position it was applied at.

Then keep stepping. The backward half has no equivalent in the other views: a kernel gradient that sums over every position, a delta travelling back through a flipped kernel, and a pool that hands its whole gradient to one cell in four.

One weight, many positions

This is the only structural difference between a convolution and a dense layer, and everything else follows from it. A dense weight gets used once per example, so its gradient is one product, $\delta_i\,a_j$. A kernel weight gets used at every position the kernel visits, 36 of them in the first layer here, so its gradient is the sum of 36 such products:

$$\frac{\partial L}{\partial K_{c,u,v}} = \sum_{i,j} \delta_{c,i,j}\;a_{c',\,i+u,\,j+v}$$

Hover any weight in $K^{(1)}$ and the whole feature map it feeds lights up. That is not decoration. Those are the terms of its gradient sum. A kernel cannot move to satisfy one position. It moves in whatever direction satisfies all 36 at once, which is why convolutional filters end up generic.

Why the backward pass flips the kernel

People state this as a rule to memorise. It is not a rule. It is what the indices say. Going forward, output $(i, j)$ reads input $(i+u,\,j+v)$. So going backward, input $(p, q)$ has to collect from every output that read it, the ones at $(p-u,\,q-v)$:

$$\delta^{(l-1)}_{c',p,q} = \sum_{c,u,v} K_{c,c',u,v}\;\delta^{(l)}_{c,\,p-u,\,q-v}$$

Addition became subtraction. Reversing the offset in a correlation is exactly what flipping the kernel does, and "full" convolution just acknowledges that border cells fed fewer outputs so they collect fewer terms. Step to the transport step and watch the corner cells collect less than the middle ones.

Where the gradient goes to die

Step to the pooling step and look at how much of that map is zero. Three cells in every four get nothing at all. Not a small number, zero, because changing them would not have changed the pooled value. Then the ReLU zeroes whatever it clipped on the way forward.

By the time the error reaches $K^{(1)}$ it is very sparse. The 36-term sum in equation (9) usually has a handful of non-zero terms, and the row list counts them for you. That is the honest picture of what a convolutional layer learns from one example: not much, from not many places.

Kinks, and why some rows are excused

ReLU and max-pooling are piecewise linear, so the loss has corners. At a corner there is no derivative. The slope coming from the left and the slope coming from the right genuinely differ, and a central difference reports the average of the two, which will not match the backward pass no matter how correct it is.

So the table above checks the two one-sided slopes separately. If they disagree by $O(1)$ the parameter is sitting on a kink, and the row gets marked and left out of the verdict instead of counted as a failure. That is not a fudge. It is the difference between "this gradient is wrong" and "this function has no gradient here". Paint a perfectly symmetric shape and you can make kinks appear on demand, usually from ties in the pooling windows.

Backpropagation is a product. Getting from the output to layer $l$ means multiplying together one weight matrix and one vector of local slopes for every layer in between. If those factors are usually smaller than one, the product shrinks geometrically.

Below is the same engine running a deliberately bad architecture: eight layers of sigmoid. The logistic sigmoid's derivative is $\sigma(z)(1 - \sigma(z))$, which never exceeds $\tfrac{1}{4}$ and is usually far less.

Step backward and watch $\|\delta^{(l)}\|$ on a log scale. Understanding this failure is worth more than watching a successful run.

Where the quarter comes from

$\sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right)$ is a downward parabola in $\sigma(z)$. It peaks at $\sigma(z) = \tfrac12$, meaning $z = 0$, where it equals $\tfrac14$. Everywhere else it is smaller, and past $|z| > 4$ it drops below $0.018$.

Each backward hop through a sigmoid layer multiplies the error by at most a quarter, before the weights get their say. Over seven layers that is a ceiling of $4^{-7} \approx 6 \times 10^{-5}$, and that is the optimistic case where every unit sits at its most responsive point.

What this cost

The early layers receive a gradient thousands of times smaller than the late ones, so under one global learning rate they effectively stop learning. The network still trains. The top few layers do all the work on top of a nearly random feature extractor.

Every standard fix attacks one factor of the product. ReLU replaces $\sigma'$ with an exact 1 on the active side. Residual connections add an identity path so the recurrence has a term that is not multiplied down. Careful initialisation keeps the weight factor near one, and normalisation keeps units out of saturation. Switch the activation below and watch which of these matters.

About

Built by Yoseph Naoom.

delta is a teaching tool. It exists because the backward pass is the part of backpropagation that gets skipped, and skipping it is what makes the whole thing feel like magic. Everything here runs in your browser, with no server and nothing to install.

The networks are deliberately tiny so that every number stays readable. There is no machine-learning library anywhere in the page. The forward and backward passes are written out by hand in about 300 lines, and every gradient is checked against finite differences while you watch.

Found a problem?

If something is wrong, unclear, or broken, I would like to hear about it. Use the form below.