Equations · Machine learning
Cross-entropy loss
The standard loss for classification: how surprised the model is by the right answer.
L = −∑ᵢ yᵢ log(ŷᵢ)
Aethel says
L equals negative the sum over i of y sub i log of y hat sub i
What it says
y is the true distribution over classes, usually one for the correct class and zero elsewhere, and ŷ is the model's predicted probabilities. With a one-hot y, the loss is minus the log of the probability the model gave to the correct class.
A confident correct prediction costs almost nothing; a confident wrong one costs a lot.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| L | L | the loss |
| yᵢ | y sub i | the true probability of class i |
| ŷᵢ | y hat sub i | the predicted probability of class i |
| log | log | the logarithm, usually natural |
Where you meet it
Training classifiers and language models; information theory.
Questions
How do you say Cross-entropy loss out loud?
L = −∑ᵢ yᵢ log(ŷᵢ) is read: “L equals negative the sum over i of y sub i log of y hat sub i.” That is exactly what Aethel says when it meets the formula in a book.
What does L mean in Cross-entropy loss?
L (said “L”) is the loss.
What does yᵢ mean in Cross-entropy loss?
yᵢ (said “y sub i”) is the true probability of class i.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.