AI Foundationsintermediate14 min · 10 steps
Gradient Descent & Optimizers
Walk downhill on the loss: how gradient descent, SGD, learning-rate choice, schedules, adaptive rates and Adam turn a gradient into better weights.
loss0.88
dataresponse
metrics are illustrativeProblemStep 1 of 10
Training is a search for the lowest loss
A model's loss measures how wrong it is on the training data — mean squared error, cross-entropy. Every weight setting w gives a loss value, so the loss is a landscape over weight space.
Here the model has a single weight and the landscape is a simple bowl. Real networks have millions or billions of weights, so we cannot try every setting — we need a way to know which way is downhill from where we stand.
← Previous lessonUnsupervised Learning: Clustering & Dimensionality ReductionNext lesson →Backpropagation & Activation Functions
Lesson 17 of 34 in AI Foundations