Equations · Machine learning
Softmax function
Turns a list of scores into probabilities that are positive and add up to one.
softmax(zᵢ) = e^(zᵢ) / ∑ⱼ e^(zⱼ)
Aethel says
softmax of z sub i equals e to the power of z sub i, over the sum over j of e to the power of z sub j
What it says
Each score is exponentiated, which makes it positive, and divided by the total of all the exponentiated scores. The largest score gets the largest share, and the shares always sum to one.
It is the last step of most classifiers and the weighting step inside attention.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| zᵢ | z sub i | the score for class i, sometimes called a logit |
| e^(zᵢ) | e to the power of z sub i | the exponential of the score |
| ∑ⱼ | the sum over j of | add up over every class j |
More on∑Summation (sigma notation)
Where you meet it
Neural-network classifiers, language models and reinforcement learning.
Questions
How do you say Softmax function out loud?
softmax(zᵢ) = e^(zᵢ) / ∑ⱼ e^(zⱼ) is read: “softmax of z sub i equals e to the power of z sub i, over the sum over j of e to the power of z sub j.” That is exactly what Aethel says when it meets the formula in a book.
What does zᵢ mean in Softmax function?
zᵢ (said “z sub i”) is the score for class i, sometimes called a logit.
What does e^(zᵢ) mean in Softmax function?
e^(zᵢ) (said “e to the power of z sub i”) is the exponential of the score.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.