Equations · Machine learning
Scaled dot-product attention
The core of the Transformer: each position mixes the values, weighted by how well its query matches each key.
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V
Aethel says
Attention of Q, K, V equals softmax of the quantity Q K transpose over the square root of d sub k, times V
What it says
Each query is compared with every key by a dot product. Softmax turns those scores into weights that sum to one, and the output is the weighted sum of the values.
Dividing by the square root of the key dimension keeps the scores from growing with the dimension, which would push the softmax into a region where its gradients are tiny.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| Q, K, V | Q, K, V | the matrices of queries, keys and values |
| Kᵀ | K transpose | the keys with rows and columns swapped |
| dₖ | d sub k | the dimension of the keys |
| softmax | softmax | turns each row of scores into weights that sum to one |
More onᵀTranspose√Square root
Where you meet it
Vaswani et al., “Attention Is All You Need” (2017), and every large language model since.
Questions
How do you say Scaled dot-product attention out loud?
Attention(Q, K, V) = softmax(QKᵀ / √dₖ)V is read: “Attention of Q, K, V equals softmax of the quantity Q K transpose over the square root of d sub k, times V.” That is exactly what Aethel says when it meets the formula in a book.
What does Q, K, V mean in Scaled dot-product attention?
Q, K, V (said “Q, K, V”) is the matrices of queries, keys and values.
What does Kᵀ mean in Scaled dot-product attention?
Kᵀ (said “K transpose”) is the keys with rows and columns swapped.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.