How does a model know what to change?
When a prediction is off, nudging every setting at random is slow. The model needs a clue about which way would reduce the mistake.
Back to the lesson shelf
Gradient
LESSON 02 The useful mistake
Gradient
descent.
A tiny compass that checks the slope, takes a step and tries again with a little less error.
In one lineFeel the slope → step downhill → repeat.
GRADIENT_DESCENT / 02 adjusting
high errorlow error
error → slope → updatestep 01 / 04
Short version: find the slope → take a step → repeat.Do not fear the mistake.
00 Overview / start here
From a wrong answer to a better one.
A model starts with a guess. Gradient descent is the repeatable adjustment loop that turns the size and direction of its error into a small next move.
A feedback loop for improving a guess.
Gradient descent measures the error, reads the local slope, and nudges the model in the direction that lowers the loss.
the basic loop
input → operation → output
-
01 / input
A current setting goes in.
The model uses its current weight or parameters to make a prediction for one example.
-
02 / operation
Measure, read, nudge.
Compare prediction with target. The gradient gives the local slope; the learning rate sets how large the step should be.
-
03 / output
A better next try.
After the update, measure the loss again. The new setting becomes the input for another round.
0.20 → 0.50, closer to 0.80.
If x = 1 and the current weight is w = 0.20, the prediction is 0.20 for a target of 0.80. With a 0.50 learning rate, the gradient is −0.60 and the next weight becomes 0.50.
01 See the landscape
Learning is a walk downhill.
Imagine the model's error as a landscape. The gradient is the local slope: it tells the model which direction should make the mistake smaller.
Move the model toward the minimum.
Drag the step slider or press the button. Notice how the loss gets smaller as the point moves into the valley.
high errorlow errorminimum
terrain generatorsmooth valley
step01/ 04
loss0.72
direction→ find slope
local slope+0.88
what the loss is sayingbig error / room to learn
model fit
28%
target fit
100%
The gradient does not know the answer. It only points toward a better next try.
02 Follow one update
One slope, one nudge.
The gradient turns a prediction error into a direction. Watch one tiny model measure its mistake, calculate the slope and move its weight.
See the mistake become movement.
Choose a starting weight, then apply one update. Every number is visible: prediction, loss, gradient and the next weight.
input x1.00one feature
weight w0.20the knob we can change
prediction ŷ0.20what the model says
target y0.80what the example says
prediction vs targeterror −0.60
The model is under-confident. The gradient will push the weight up.
parameter spacemove right / increase w
too low
too high
0.00.40.81.3
target 0.80
now 0.20
after step 0.50
gradient −0.60
loss / half squared error0.18smaller is better
gradient ∂L / ∂w−0.60negative → increase w
next weight0.50w − rate × gradient
gradient = 0.20 − 0.80 = −0.60 → next w = 0.20 − 0.50 × (−0.60) = 0.50
choose a starting point
A gradient is useful because it makes a mistake actionable.
03 Tune the step size
Too timid or too wild?
The learning rate controls how far the model moves each time. Tiny steps are slow. Giant steps can jump over the valley.
Find a comfortable pace.
Drag the learning rate, then compare three training runs. Click a path to copy its pace into the slider.
learning rate0.50
slowbalancedchaos
try a pace
Balanced steps: fast enough to learn, gentle enough to stay in the valley.
four updates from the same starting pointbalanced / settles quickly
next movetoward the valley
after 4 steps0.00 loss
lessonthe step size matters
Good training is a rhythm: measure, adjust, try again.
03 / check your mental model
What does the gradient tell the model?
At the current point, the gradient is mostly a…
Pick one. The landscape will tell you if the picture clicks.
FAQ Gradient descent / quick answers
Take the
next step.
Use these compact answers to connect the landscape to the training loop.
01 / definitionWhat is gradient descent?
Gradient descent is an optimization loop: measure the current error, read the local slope, and move the model’s parameters in the direction that should reduce the error.
02 / paceWhat does the learning rate control?
The learning rate controls the size of each update. Small steps can be steady but slow; large steps can learn quickly or jump past a useful low point.
03 / failure modeWhy can a training step overshoot?
A large update treats the local direction as if it were safe for a long distance. If the step is too big, the parameter can cross the valley and make the next error larger.