Equations · Machine learning
Evidence lower bound (ELBO)
The objective of variational autoencoders: a lower bound on how well a model explains the data.
log p(x) ≥ E_q[log p(x | z)] − D_KL(q(z | x) ‖ p(z))
Aethel says
log p of x, is greater than or equal to the expected value under q of log p of x given z, minus the KL divergence of q of z given x, from p of z
What it says
The log probability of the data cannot be computed directly, but this bound can. Its first term rewards reconstructing x from the latent code z; the second penalises an encoder distribution q that strays from the prior p(z).
Raising the bound trains the encoder and the decoder together.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| p(x) | p of x | the probability the model gives to the data, the evidence |
| E_q[…] | the expected value under q of | the average over codes z drawn from q |
| q(z | x) | q of z given x | the encoder: a distribution over codes for input x |
| D_KL | the KL divergence | how far q is from the prior p(z) |
More on≥Greater than or equal to|Vertical bar: given, divides
Where you meet it
Variational inference and generative models.
Questions
How do you say Evidence lower bound out loud?
log p(x) ≥ E_q[log p(x | z)] − D_KL(q(z | x) ‖ p(z)) is read: “log p of x, is greater than or equal to the expected value under q of log p of x given z, minus the KL divergence of q of z given x, from p of z.” That is exactly what Aethel says when it meets the formula in a book.
What does p(x) mean in Evidence lower bound?
p(x) (said “p of x”) is the probability the model gives to the data, the evidence.
What does E_q[…] mean in Evidence lower bound?
E_q[…] (said “the expected value under q of”) is the average over codes z drawn from q.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.