Equations · Machine learning
Kullback–Leibler divergence
How different one probability distribution is from another, measured in lost information.
D_KL(P ‖ Q) = ∑ₓ P(x) log(P(x) / Q(x))
Aethel says
the KL divergence of P from Q equals the sum over x of P of x, times log of the quantity P of x, over Q of x
What it says
It is the expected extra cost, in bits or nats, of coding samples from P with a code designed for Q. It is zero exactly when the two distributions are the same, and positive otherwise.
It is not a distance: swapping P and Q usually gives a different value.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| D_KL(P ‖ Q) | the KL divergence of P from Q | the divergence of distribution P from reference distribution Q |
| P(x), Q(x) | P of x, Q of x | the probabilities the two distributions give to outcome x |
| ∑ₓ | the sum over x of | add up over every outcome x |
More on∑Summation (sigma notation)
Where you meet it
Information theory, variational inference and the training of generative models.
Questions
How do you say Kullback–Leibler divergence out loud?
D_KL(P ‖ Q) = ∑ₓ P(x) log(P(x) / Q(x)) is read: “the KL divergence of P from Q equals the sum over x of P of x, times log of the quantity P of x, over Q of x.” That is exactly what Aethel says when it meets the formula in a book.
What does D_KL(P ‖ Q) mean in Kullback–Leibler divergence?
D_KL(P ‖ Q) (said “the KL divergence of P from Q”) is the divergence of distribution P from reference distribution Q.
What does P(x), Q(x) mean in Kullback–Leibler divergence?
P(x), Q(x) (said “P of x, Q of x”) is the probabilities the two distributions give to outcome x.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.