Equations · Machine learning
Gradient descent update
One step of the method that trains most machine-learning models: move the parameters downhill on the loss.
θ ← θ − η ∇L(θ)
Aethel says
theta is updated to theta minus eta times the gradient of L of theta
What it says
The gradient of the loss points in the direction in which the loss rises fastest. Subtracting a small multiple of it moves the parameters a little way downhill. Repeated many times, the loss falls.
The learning rate η sets the step size: too large and training diverges, too small and it crawls.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| θ | theta | the model's parameters |
| ← | is updated to | replace the left side with the right side |
| η | eta | the learning rate |
| ∇L(θ) | the gradient of L of theta | the gradient of the loss with respect to the parameters |
More on∇Nabla (del)θThetaηEta
Where you meet it
Machine-learning and optimisation courses.
Questions
How do you say Gradient descent update out loud?
θ ← θ − η ∇L(θ) is read: “theta is updated to theta minus eta times the gradient of L of theta.” That is exactly what Aethel says when it meets the formula in a book.
What does θ mean in Gradient descent update?
θ (said “theta”) is the model's parameters.
What does ← mean in Gradient descent update?
← (said “is updated to”) is replace the left side with the right side.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.