See how a network learns.
Change the inputs. Follow the calculations. Watch the gradients.
Inside a prediction2 → 3 → 2
← → to step · Space to play · Hover a symbol to trace its connections.
Change the inputs. Follow the calculations. Watch the gradients.
← → to step · Space to play · Hover a symbol to trace its connections.
Follow two inputs through a small neural network.
A model uses adjustable parameters to make a prediction. Each unit multiplies its inputs by weights, adds a bias, then applies an activation function. These units form layers.
The forward pass computes the output. This untrained network has 17 parameters.
Make a prediction, measure the loss, then work backward.
The shaded field shows the prediction for each input. The decision boundary separates the two classes. Select a point to inspect it.
Choose XOR and train the network. A hidden layer can separate the four points; the linear model cannot. At equal probabilities, a point is marked undecided.
Backpropagation computes the gradient: a partial derivative of the loss for each parameter. The error signal $\delta$ passes through the transposed weights, then multiplies by the activation function's slope.
Gradient descent subtracts a multiple of each gradient from its parameter. The learning rate controls the update size.
With softmax and cross-entropy, the output error is $\hat{y}-y$. The Jacobian calculation below gives the same result.
Gradient checking nudges each parameter by $h$ and compares the measured loss change with the backward pass.
Each row is one sample. Their gradients add together.
A batch stacks samples into a matrix. Activations gain a row per sample. Weight and bias dimensions stay fixed.
The transpose aligns the dimensions for each matrix product. Biases are added to every row.
The batched loss here is a sum. Each weight gradient is a dot product across the samples.
Each row is computed independently, so the batch Jacobian is block diagonal. Layers such as batch normalisation would couple the rows and change this property.
The trace trick rearranges a scalar differential to identify the gradient. A directional derivative checks the result.
Compare three calculations of the same gradient, then check that changing one sample leaves the other outputs unchanged.
Draw on the input. Step through the filters and back again.
A convolution applies one kernel at many positions. Each output channel is a feature map. Weight sharing means a filter's gradient sums its contributions across the image.
The second layer combines channels. Its 12 outputs are flattened into a vector for the final layer.
ReLU keeps positive values and sets negative values to zero. Max-pooling keeps the largest value in each 2×2 patch; its gradient returns to that cell.
A ReLU at zero or a max-pool tie can create a kink. The numerical check marks these points separately because a unique derivative may not exist.
Gradient checking compares the backward pass with finite differences. Kinks are excluded from the verdict.
Compare sigmoid, tanh, and ReLU across eight layers.
Each backward step multiplies by the weights and the activation function's derivative. Sigmoid's slope never exceeds $0.25$, which can make the error signal much smaller in earlier layers.
Tanh has a slope up to $1$. ReLU's slope is $1$ for positive inputs and $0$ for negative inputs. Neither guarantees a constant gradient: the weights and inactive units also matter.
A vanishing gradient gives earlier layers very small updates during training.
Look up a term, then find it in a simulation.
Delta is an interactive guide to backpropagation, built by Yoseph Naoom.
The networks are small enough to inspect every value. All calculations run in your browser. No account, uploads, or external model service.
Include the simulation, what you changed, and what you expected.