AI Foundationsintermediate14 min · 11 steps
Backpropagation & Activation Functions
How a network learns: a forward pass computes the loss, backpropagation sends blame back with the chain rule, and non-linear activations — sigmoid, tanh, ReLU, softmax — make depth worth having.
loss2.3
dataerror
metrics are illustrativeProblemStep 1 of 11
Millions of weights, one loss number
A network makes a prediction, compares it with the label and gets a single loss value. To improve, gradient descent needs ∂L/∂w — how the loss changes when each individual weight changes — for every weight in every layer.
Nudging each weight separately and re-running the network would take one forward pass per weight: hopeless for millions of weights.
Lesson 18 of 34 in AI Foundations