Skip to content
anythingai.space
AI Foundationsintermediate14 min · 11 steps

Backpropagation & Activation Functions

How a network learns: a forward pass computes the loss, backpropagation sends blame back with the chain rule, and non-linear activations — sigmoid, tanh, ReLU, softmax — make depth worth having.

loss2.3
Which of the weights in W₁, W₂, W₃ caused this error— and by how much?Input xpixels / featuresLayer 1W₁, φLayer 2W₂, φOutput ŷW₃, softmaxLossvs label y2.3
dataerror
metrics are illustrative
1 / 11
ProblemStep 1 of 11

Millions of weights, one loss number

A network makes a prediction, compares it with the label and gets a single loss value. To improve, gradient descent needs ∂L/∂w — how the loss changes when each individual weight changes — for every weight in every layer.

Nudging each weight separately and re-running the network would take one forward pass per weight: hopeless for millions of weights.

Lesson 18 of 34 in AI Foundations