Equations · Machine learning
Bellman optimality equation
The value of a state is the best immediate reward plus the discounted value of where you land next.
V(s) = maxₐ (R(s, a) + γ ∑ₛ′ P(s′ | s, a) V(s′))
Aethel says
V of s equals the maximum over a of the quantity R of s, a, plus gamma the sum over s prime of P of s prime given s, a, times V of s prime
What it says
In a decision problem, the value of being in state s is found by trying each action a: take its reward, add the value of the states it can lead to, weighted by their probabilities and discounted by γ, and keep the best action.
Value iteration and Q-learning solve this equation step by step.
Symbol by symbol
| Symbol | Say it | Means |
|---|---|---|
| V(s) | V of s | the value of state s |
| maxₐ | the maximum over a of | the best result over all actions a |
| R(s, a) | R of s, a | the reward for taking action a in state s |
| γ | gamma | the discount factor, between 0 and 1 |
| P(s′ | s, a) | P of s prime given s, a | the probability of moving to state s′ |
Where you meet it
Reinforcement learning, operations research and economics.
Questions
How do you say Bellman optimality equation out loud?
V(s) = maxₐ (R(s, a) + γ ∑ₛ′ P(s′ | s, a) V(s′)) is read: “V of s equals the maximum over a of the quantity R of s, a, plus gamma the sum over s prime of P of s prime given s, a, times V of s prime.” That is exactly what Aethel says when it meets the formula in a book.
What does V(s) mean in Bellman optimality equation?
V(s) (said “V of s”) is the value of state s.
What does maxₐ mean in Bellman optimality equation?
maxₐ (said “the maximum over a of”) is the best result over all actions a.
Can I listen to the equations in my own textbook like this?
Yes. Upload a PDF, EPUB or text file and Aethel reads it aloud with every equation spoken as maths, the way it reads this one, and highlights each sentence as it goes. You can start free.