Dynamic Programming

Reinforcement Learning · Lecture 2
Claire Vernade
University of Technology Nuremberg
22 October 2026

Today

  1. Recap & notation — and our running example: the two-exit maze
  2. Value functions — how good is a policy?
  3. Bellman equations — expectation, optimality, operators
  4. Dynamic programming — policy evaluation, policy iteration, value iteration
Today's assumption The MDP is known: we can query $P$ and $r$.
This is planning, not yet learning.
Part 1

Recap & running example

Notation, the maze we will use all lecture, and the objective.

RecapAgent and environment

Agent Environment A t S t+1 R t+1

At each time step $t = 0, 1, 2, \dots$

  • the agent observes the state $S_t \in \S$
  • picks an action $A_t \in \A$
  • receives a reward $R_{t+1}$ and moves to $S_{t+1}$

Trajectory: $S_0, A_0, R_1, S_1, A_1, R_2, S_2, \dots$

RecapMarkov Decision Process

Definition A (discounted) MDP is a tuple $\M = (\S, \A, P, r, \gamma)$:
  • $\S$ finite set of states, $\A$ finite set of actions
  • $P(s' \mid s, a)$ transition probabilities  — Markov: depends only on $(s,a)$
  • $r(s,a)$ expected immediate reward, $\;r(s,a) = \E[R_{t+1} \mid S_t = s, A_t = a]$
  • $\gamma \in (0,1)$ discount factor

An MDP is a model of the problem. It assumes the state is fully observed, the dynamics are Markov, and the goal can be written as a cumulative reward (the "reward hypothesis").

Running exampleThe two-exit maze

  • 12 states $s_1, \dots, s_{12}$, start at $s_1$
  • Actions: $\A = \{\uparrow, \rightarrow, \downarrow, \leftarrow\}$
  • Deterministic moves; walls and borders: stay put
  • Two exits end the episode:
    • small exit: $+1$, close by
    • big exit: $+5$, far away
  • All other rewards are $0$
Discuss with your neighbour Which exit should the agent go to?

Running exampleThe maze as an MDP

  • $\S = \{s_1, \dots, s_{12}\}$ + two terminal exits. $\;r(s,a) = 5$ or $1$ when $a$ enters an exit, else $0$
  • Deterministic: $P(s' \mid s,a) \in \{0,1\}$, e.g. $P(s_7 \mid s_6, \rightarrow) = 1$, $\;P(s_1 \mid s_1, \leftarrow) = 1$
Variant: the slippery maze W.p. $0.8$ the intended move; w.p. $0.2$ a slip to one of the 3 other neighbours ($0.2/3$ each). Slipping into a wall = staying put.
  • Stochastic transitions: same state, same action, several possible next states
  • Model external perturbations: wind, slippery floor, actuator noise, other agents
  • $r(s,a)$ is now an expectation: $r(s_6,\rightarrow) = \tfrac{0.2}{3}\cdot 1 \approx 0.07$
$P(\cdot \mid s_6, \rightarrow)$. Toggle to compare.

Running exampleIn the computer: $P$ and $r$ are arrays

S, A = 14, 4   # 12 cells + 2 exits; ↑ → ↓ ←
P = np.zeros((A, S, S))  # P[a, s, s']
r = np.zeros((S, A))     # r[s, a]

# every row is a distribution
assert np.allclose(P.sum(axis=2), 1)

# one Bellman backup, all (s, a) at once
Q = r + gamma * np.einsum("ast,t->sa", P, V)
V = Q.max(axis=1)
  • $P$ = $|\A| = 4$ stochastic matrices of size $|\S| \times |\S|$
  • Walls are not states. The exits are absorbing: $P(G \mid G, a) = 1$, $r = 0$
  • Memory $|\A|\,|\S|^2$: 784 numbers here; impossible for chess or Go

Policies

Definition A (stationary, Markov) policy maps states to distributions over actions: $\;\pi(a \mid s) = \P(A_t = a \mid S_t = s)$.
Deterministic policy: $\pi : \S \to \A$.
$\pi_{\text{small}}$: shortest path to $+1$
$\pi_{\text{big}}$: shortest path to $+5$
$\pi_{\text{unif}}$: uniformly random

The return

Definition The (discounted) return from time $t$: $$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$

Following $\pi_{\text{small}}$ from $s_1$

$s_1 \xrightarrow{\uparrow} s_5 \xrightarrow{\rightarrow} \text{small exit}$

$G_0 = 0 + \gamma \cdot 1 = \gamma$

Following $\pi_{\text{big}}$ from $s_1$

$s_1 \to s_2 \to s_3 \to s_6 \to s_7 \to s_{10} \to \text{big exit}$

$G_0 = 0 + \dots + 0 + \gamma^5 \cdot 5$

Let's practice: when is the big exit worth it?

From $s_1$, compare $\;\gamma\;$ (small exit) and $\;5\gamma^5\;$ (big exit).

$$5\gamma^5 > \gamma \iff \gamma^4 > \tfrac15 \iff \gamma > 5^{-1/4} \approx 0.669$$

Think Is the threshold the same for every starting state? (Try $s_6$, next to both routes.)

Why discount?

  • Keeps returns finite: if $|R_t| \le R_{\max}$ then $|G_t| \le \frac{R_{\max}}{1-\gamma}$
  • Expresses uncertainty about the future (→)
  • Effective horizon $\E[T] = \frac{1}{1-\gamma}$: $\gamma = 0.9 \Rightarrow 10$ steps, $\;\gamma = 0.99 \Rightarrow 100$
  • Humans and animals prefer immediate rewards
Take-away $\gamma$ is part of the problem, not of the algorithm. Changing $\gamma$ changes what "optimal" means.
Discounting = a geometric belief about the end

Suppose the episode ends after each step w.p. $1-\gamma$: a random horizon $T$ with $\P(T > k) = \gamma^k$. The expected undiscounted reward until $T$ is

$$\E_T\Big[\sum_{k=0}^{T-1} R_{t+k+1}\Big] = \sum_{k \ge 0} \P(T > k)\, R_{t+k+1} = \sum_{k \ge 0} \gamma^k R_{t+k+1} = G_t$$

A reward $k$ steps ahead counts with weight $\gamma^k$ = the probability of still being there.

Beyond geometric If we are uncertain about the stopping rate itself, the weights become hyperbolic, $\propto 1/(1+\kappa k)$ (Fedus et al., Hyperbolic discounting and learning over multiple horizons, arXiv:1902.06865). This raises additional computational questions (no single Bellman recursion); out of scope here.

The objective

Goal Find a policy $\pi$ maximising the expected discounted return $$\max_\pi \; \E_\pi\Big[\sum_{t=0}^{\infty} \gamma^t R_{t+1} \;\Big|\; S_0 = s\Big] \quad \text{for every } s$$

Prediction

Given $\pi$: how good is it? → evaluate $\pi$

Control

Find the best $\pi$ → optimise

When the MDP is known, this is an optimisation problem (planning). When it is unknown, it's a learning problem (next lectures).

Part 2

Value functions

How good is it to be in a state, or to take an action, when following $\pi$?

State-value function

Definition The value of state $s$ under policy $\pi$ is the expected return when starting in $s$ and following $\pi$: $$\Vpi(s) = \E_\pi\big[\,G_t \mid S_t = s\,\big]$$

Action-value function

Definition First take action $a$ in $s$, then follow $\pi$: $$\Qpi(s,a) = \E_\pi\big[\,G_t \mid S_t = s, A_t = a\,\big]$$

The two are linked:

$$\Vpi(s) = \sum_{a} \pi(a \mid s)\, \Qpi(s,a)$$
$$\Qpi(s,a) = r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, \Vpi(s')$$

Maze$\Vpi$ of the uniformly random policy, $\gamma = 0.9$

Fix $\pi$ = uniformly random: each action w.p. $\tfrac14$.
I computed $\Vpi$ for you (left).

  • Why is $s_{10}$ the most valuable state?
  • Why is $V(s_5) > V(s_1)$?
  • The random walker often bumps into walls; it can still stumble into the big exit.
Open questions
  • How did I compute these numbers? (Parts 3 and 4)
  • Can I improve the policy using this function? (Part 4)

Let's practice: value of $\pi_{\text{small}}$

$\gamma = 0.9$. Policy is deterministic, moves are deterministic: the return is a single number.

If the exit is $d$ steps away: $\;\Vpi(s) = \gamma^{d-1}\cdot 1$.

Click a cell to reveal its value.

Think Do you need to compute each value independently of all the others, already computed ones?

No: e.g. $\Vpi(s_1) = 0 + \gamma\, \Vpi(s_5) = 0.9 \cdot 1$. Each value is the reward plus $\gamma \times$ the value of the next state. That is the Bellman equation.

Part 3

Bellman equations

Value = immediate reward + discounted value of where you land.

Derivation 1/2Bellman expectation equation

Recall the return: $\;G_t = \sum_{k \ge 0} \gamma^k R_{t+k+1} = R_{t+1} + \gamma \big(R_{t+2} + \gamma R_{t+3} + \dots\big) = R_{t+1} + \gamma\, G_{t+1}$

Fix a policy $\cpol{\pi}$ (e.g. $\pi_{\text{big}}$). Goal: compute $\Vpi(s) = \E_\pi[G_t \mid S_t = s]$ for all $s$, efficiently.

$$\Vpi(s) = \E_\pi[\crew{R_{t+1}} + \gamma\, \cfut{G_{t+1}} \mid S_t = s] = \underbrace{\E_\pi[\crew{R_{t+1}} \mid S_t = s]}_{\crew{\text{immediate reward}}} + \gamma\, \underbrace{\E_\pi[\cfut{G_{t+1}} \mid S_t = s]}_{\cfut{\text{future}}}$$
Why the future term is $\E_\pi[\Vpi(S_{t+1}) \mid S_t = s]$

Condition on where we land, then use the Markov property: once $S_{t+1} = s'$ is known, what happens next does not depend on the past.

$$\E_\pi[\cfut{G_{t+1}} \mid S_t = s] = \E_\pi\big[\, \E_\pi[\cfut{G_{t+1}} \mid S_{t+1}] \;\big|\; S_t = s \big] = \E_\pi[\cfut{\Vpi(S_{t+1})} \mid S_t = s]$$

Careful: $G_{t+1} \neq \Vpi(S_{t+1})$ as random variables. They are only equal in expectation given $S_{t+1}$ (tower rule).

Derivation 2/2Bellman expectation equation

Develop each term: first draw $a \sim \cpol{\pi(\cdot \mid s)}$, then $s' \sim \cfut{P(\cdot \mid s,a)}$.

Immediate reward
$$\E_\pi[\crew{R_{t+1}} \mid S_t = s] = \sum_a \cpol{\pi(a \mid s)}\, \crew{r(s,a)}$$
Future
$$\E_\pi[\cfut{\Vpi(S_{t+1})} \mid S_t = s] = \sum_a \cpol{\pi(a \mid s)} \sum_{s'} \cfut{P(s' \mid s,a)\, \Vpi(s')}$$
Recombine: Bellman expectation equation
$$\Vpi(s) = \sum_a \cpol{\pi(a \mid s)} \Big( \crew{r(s,a)} + \gamma \sum_{s'} \cfut{P(s' \mid s,a)\, \Vpi(s')} \Big)$$

One equation per state: $|\S|$ linear equations in the $|\S|$ unknowns $\Vpi(s)$.

Same for $Q$: $\;\Qpi(s,a) = \crew{r(s,a)} + \gamma \sum_{s'} \cfut{P(s' \mid s,a)} \sum_{a'} \cpol{\pi(a' \mid s')}\, \Qpi(s',a')$

Backup diagram

s s′ V^π(s′) π(a|s) a r(s,a) P(s′|s,a)
$$\Vpi(s) = \sum_a \cpol{\pi(a \mid s)} \Big( \crew{r(s,a)} + \gamma \sum_{s'} \cfut{P(s' \mid s,a)\, \Vpi(s')} \Big)$$
  • open circle = state, filled = action
  • top level: average over actions with $\cpol{\pi}$
  • at the action: collect the reward $\crew{r(s,a)}$
  • bottom level: average over next states with $\cfut{P}$, weighting their values $\cfut{\Vpi(s')}$
  • "back up" the leaves to the root: that is one Bellman update

Let's practice: Bellman equation in the maze

Uniformly random policy, $\gamma = 0.9$. Write the equation for $s_{10}$:

$\Vpi(s_{10}) = \tfrac14\big[(5 + 0) + \gamma \Vpi(s_{10}) + \gamma \Vpi(s_7) + \gamma\Vpi(s_{10})\big]$

↑ exit, → wall, ↓ $s_7$, ← wall. Click any state to check.

Matrix form and closed-form solution

Stack everything into vectors: $r^\pi(s) = \sum_a \pi(a|s) r(s,a)$, $\;P^\pi(s,s') = \sum_a \pi(a|s) P(s'|s,a)$.

$$\Vpi = r^\pi + \gamma P^\pi \Vpi \quad\Longrightarrow\quad \Vpi = (I - \gamma P^\pi)^{-1} r^\pi$$

Bellman operator

Definition For any $V \in \R^{|\S|}$, define $\Tpi V \in \R^{|\S|}$ by $$(\Tpi V)(s) = \sum_a \pi(a \mid s)\Big( r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, V(s') \Big)$$

The Bellman equation says: $\Vpi$ is a fixed point, $\;\Tpi \Vpi = \Vpi$.

Proposition (contraction) $\norm{\Tpi U - \Tpi V}_\infty \le \gamma \norm{U - V}_\infty$ for all $U, V$.
By Banach's fixed-point theorem, $\Vpi$ is the unique fixed point, and $V_{k+1} = \Tpi V_k$ converges to it from any $V_0$.

Optimal value functions

Definition $$\Vstar(s) = \max_\pi \Vpi(s), \qquad \Qstar(s,a) = \max_\pi \Qpi(s,a)$$ A policy $\pistar$ is optimal if $V^{\pistar}(s) = \Vstar(s)$ for all $s$.
Discuss $\Vstar(s)$ is a maximum taken separately for each state. Could the best policy when starting from $s_1$ differ from the best one when starting from $s_6$? Is there a single policy that is best everywhere at once?

Not obvious a priori. For MDPs the answer is yes: see the next slide.

Bellman optimality equation

$$\Vstar(s) = \max_{a} \Big( r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, \Vstar(s') \Big) = \max_a \Qstar(s,a)$$
Optimal Bellman operator $(\Tstar V)(s) = \max_a \big( r(s,a) + \gamma \sum_{s'} P(s' \mid s,a) V(s') \big)$  — also a $\gamma$-contraction.

Fundamental theorem of MDPs

Theorem For a finite discounted MDP:
  1. There is a unique $\Vstar$ with $\Tstar \Vstar = \Vstar$.
  2. There exists an optimal policy that is stationary and deterministic.
  3. A policy is optimal iff it is greedy w.r.t. $\Qstar$: $\;\pistar(s) \in \argmax_a \Qstar(s,a)$.

So we only need to compute $\Vstar$ (or $\Qstar$); acting greedily then gives an optimal policy.

Maze$\Vstar$ and $\pistar$ depend on $\gamma$

$\gamma = 0.9$
$\gamma = 0.5$

MazePlay with $\gamma$

Questions?

5-minute break

Part 4

Dynamic programming

Turn the Bellman equations into algorithms.

What is dynamic programming?

RequiresFull knowledge of the MDP: $P$ and $r$.

Policy evaluation (prediction)

Problem: given $\pi$, compute $\Vpi$. Idea: iterate the Bellman operator.

Iterative policy evaluation
  1. Initialise $V_0(s) = 0$ for all $s$
  2. For $k = 0, 1, 2, \dots$:   $V_{k+1} = \Tpi V_k$, i.e. for every $s$, $$V_{k+1}(s) = \sum_a \pi(a|s)\Big(r(s,a) + \gamma \sum_{s'} P(s'|s,a)\, V_k(s')\Big)$$
  3. Stop when $\norm{V_{k+1} - V_k}_\infty$ is small

Convergence: $\norm{V_k - \Vpi}_\infty \le \gamma^k \norm{V_0 - \Vpi}_\infty$, by contraction.

Let's practice: policy evaluation by hand

$\pi_{\text{unif}}$, $\gamma = 0.9$, $V_0 = 0$.

  1. Compute $V_1$: which states change? (Only neighbours of an exit.)
  2. Compute $V_2(s_7)$ and $V_2(s_{10})$ by hand.
  3. Press Step to check; click a state for its backup.

Policy improvement

Given $\Vpi$, act greedily: $$\pi'(s) \in \argmax_a \Qpi(s,a) = \argmax_a \Big( r(s,a) + \gamma \sum_{s'} P(s'|s,a)\, \Vpi(s') \Big)$$

Policy improvement theorem If $\Qpi(s, \pi'(s)) \ge \Vpi(s)$ for all $s$, then $V^{\pi'}(s) \ge \Vpi(s)$ for all $s$.

If the improvement stops ($\pi' = \pi$), then $\Vpi$ satisfies the Bellman optimality equation, so $\pi$ is optimal.

Proof: see the self-study section at the end.

Policy iteration

Policy iteration
  1. Start from any policy $\pi_0$
  2. Evaluate: compute $V^{\pi_k}$ (exactly, or by iterative policy evaluation)
  3. Improve: $\pi_{k+1} = \text{greedy}(V^{\pi_k})$
  4. Repeat until $\pi_{k+1} = \pi_k$

$\pi_0 \xrightarrow{E} V^{\pi_0} \xrightarrow{I} \pi_1 \xrightarrow{E} V^{\pi_1} \xrightarrow{I} \cdots \to \pistar$

Terminates in finitely many steps: there are finitely many deterministic policies, and each step strictly improves until optimal.

MazePolicy iteration from the random policy

Value iteration

Idea: skip full evaluation and iterate the optimal Bellman operator directly.

Value iteration
  1. Initialise $V_0(s) = 0$ for all $s$
  2. For $k = 0, 1, 2, \dots$:   $V_{k+1} = \Tstar V_k$, i.e. $$V_{k+1}(s) = \max_a \Big(r(s,a) + \gamma \sum_{s'} P(s'|s,a)\, V_k(s')\Big)$$
  3. Return the greedy policy w.r.t. the final $V_k$

Converges to $\Vstar$ at rate $\gamma^k$. You can view it as policy iteration with one sweep of evaluation per improvement.

Let's practice: value iteration by hand

$\gamma = 0.9$, $V_0 = 0$. Compute $V_1, V_2, V_3$.

  • $V_1$: $1$ next to the small exit, $5$ at $s_{10}$, $0$ elsewhere
  • $V_2(s_7) = \max(\dots, 0 + 0.9 \cdot 5) = 4.5$
  • When does $s_1$ first "see" the big exit? After how many iterations is its greedy action final?

Value iteration vs policy iteration

Value iteration Policy iteration
Per iteration one sweep of $\Tstar$: $O(|\S|^2|\A|)$ full evaluation $O(|\S|^3)$ + improvement
Iterations many (rate $\gamma^k$) few (finite, often very few)
Intermediate $V_k$ not the value of any policy always $V^{\pi_k}$
Stopping rule for VI If $\norm{V_{k+1} - V_k}_\infty < \varepsilon\frac{1-\gamma}{2\gamma}$, the greedy policy w.r.t. $V_{k+1}$ is $\varepsilon$-optimal.

Puterman (1994), Thm 6.3.1. Proof and discussion: self-study section at the end.

OptionalGeneralised policy iteration

π V evaluation: V → V π improvement: π → greedy(V)
  • Any interleaving of (partial) evaluation and (partial) improvement
  • VI and PI are the two extremes
  • Almost every RL algorithm in this course is GPI with approximate evaluation: MC, TD, SARSA, Q-learning, actor-critic

Summary

Problem Bellman equation Algorithm
Prediction Expectation: $\Vpi = \Tpi \Vpi$ Iterative policy evaluation
Control Expectation + greedy improvement Policy iteration
Control Optimality: $\Vstar = \Tstar \Vstar$ Value iteration

Next time: Model-free prediction

What if we don't know $P$ and $r$, and can only sample trajectories in the maze?

Same maze, same $\gamma$. We will compare with the exact values computed today.

Self-study

Proofs

Not covered in class. Everything follows from two properties of the Bellman operators: monotonicity and contraction.

Self-studyTwo tools

Inequalities between value functions are pointwise: $U \le V$ means $U(s) \le V(s)$ for all $s$.

Lemma 1: monotonicity If $U \le V$, then $\Tpi U \le \Tpi V$ and $\Tstar U \le \Tstar V$.

Proof: $(\Tpi V)(s) = \sum_a \pi(a|s)\big(r(s,a) + \gamma \sum_{s'} P(s'|s,a) V(s')\big)$ only adds up $V$ with non-negative weights. For $\Tstar$, a maximum of larger terms is larger.

Lemma 2: contraction (from the lecture) $\norm{\Tpi U - \Tpi V}_\infty \le \gamma \norm{U - V}_\infty$, and the same for $\Tstar$. Hence $(\Tpi)^k V \to \Vpi$ and $(\Tstar)^k V \to \Vstar$ for any $V$.

For $\Tstar$, use $|\max_a f(a) - \max_a g(a)| \le \max_a |f(a) - g(a)|$.

Greedy policies If $\pi'$ is greedy w.r.t. $V$, i.e. $\pi'(s) \in \argmax_a \big(r(s,a) + \gamma \sum_{s'} P(s'|s,a) V(s')\big)$, then $$T^{\pi'} V = \Tstar V \;\ge\; \Tpi V \quad \text{for every policy } \pi.$$

Self-studyProof of the policy improvement theorem

Let $\pi'$ be greedy w.r.t. $\Vpi$. (More generally, assume $\Qpi(s,\pi'(s)) \ge \Vpi(s)$ for all $s$, i.e. $T^{\pi'}\Vpi \ge \Vpi$.)

$$T^{\pi'} \Vpi \;=\; \Tstar \Vpi \;\ge\; \Tpi \Vpi \;=\; \Vpi$$

Apply $T^{\pi'}$ on both sides and use monotonicity (Lemma 1), again and again:

$$\Vpi \;\le\; T^{\pi'}\Vpi \;\le\; (T^{\pi'})^2 \Vpi \;\le\; \cdots \;\le\; (T^{\pi'})^k \Vpi \;\xrightarrow[k \to \infty]{}\; V^{\pi'} \quad \text{(Lemma 2)}$$

Hence $V^{\pi'} \ge \Vpi$. $\quad\blacksquare$

Same proof, unrolled ("telescoping", as in Sutton & Barto §4.2) $\Vpi(s) \le \Qpi(s, \pi'(s)) = \E_{\pi'}[R_{t+1} + \gamma \Vpi(S_{t+1}) \mid S_t = s] \le \E_{\pi'}[R_{t+1} + \gamma R_{t+2} + \gamma^2 \Vpi(S_{t+2}) \mid S_t = s] \le \cdots \le V^{\pi'}(s)$

Self-studyPolicy iteration ends with an optimal policy

Let $\pi_{k+1}$ be greedy w.r.t. $V^{\pi_k}$. By the improvement theorem, $V^{\pi_{k+1}} \ge V^{\pi_k}$.

If there is no improvement, the policy is optimal Suppose $V^{\pi_{k+1}} = V^{\pi_k}$. Then $$V^{\pi_k} = V^{\pi_{k+1}} = T^{\pi_{k+1}} V^{\pi_{k+1}} = T^{\pi_{k+1}} V^{\pi_k} = \Tstar V^{\pi_k}$$ so $V^{\pi_k}$ is the fixed point of $\Tstar$: $V^{\pi_k} = \Vstar$ and $\pi_k$ is optimal.
Otherwise, it strictly improves, so it stops If $V^{\pi_{k+1}} \ne V^{\pi_k}$, then $V^{\pi_{k+1}} \ge V^{\pi_k}$ with a strict inequality in some state. The sequence of value functions never repeats, and there are only $|\A|^{|\S|}$ deterministic policies. So policy iteration stops after finitely many steps, at an optimal policy. $\quad\blacksquare$

In practice: break ties consistently (keep the current action if it is still greedy) and stop when the policy no longer changes.

Self-studyWhen can value iteration stop?

Theorem (Puterman 1994, Thm 6.3.1) If $\norm{V_{k+1} - V_k}_\infty < \varepsilon \frac{1-\gamma}{2\gamma}$, and $\pi$ is greedy w.r.t. $V_{k+1}$, then $\;\norm{V^\pi - \Vstar}_\infty \le \varepsilon$: the policy is $\varepsilon$-optimal.

What it says

  • A guarantee on the loss of the policy we output, measured on its true value $V^\pi$
  • Computable: only uses two consecutive iterates

What it does not say

  • That $\pi$ is exactly optimal, or that the greedy policy will not change any more
  • Finitely many policies, so a small enough $\varepsilon$ gives exact optimality, but that threshold depends on unknown value gaps
In the maze ($\gamma = 0.9$, $\varepsilon = 0.1$, threshold $\approx 0.0056$) Deterministic: the rule stops at $k = 11$; the greedy policy was already optimal from $k = 8$.
Slippery: stops at $k = 20$; the greedy policy was optimal from $k = 13$. The rule is conservative.

Self-studyProof of the stopping rule

Write $V' = V_{k+1} = \Tstar V_k$, $\;\Delta = \norm{V' - V_k}_\infty$, and $\pi$ greedy w.r.t. $V'$, so $\Tpi V' = \Tstar V'$. Split the error: $\;\norm{V^\pi - \Vstar} \le \norm{V^\pi - V'} + \norm{V' - \Vstar}$.

(1) The estimate is close to $\Vstar$ $\norm{V' - \Vstar} = \norm{\Tstar V_k - \Tstar \Vstar} \le \gamma \norm{V_k - \Vstar} \le \gamma\big(\Delta + \norm{V' - \Vstar}\big)$ $\;\Rightarrow\; \norm{V' - \Vstar} \le \dfrac{\gamma}{1-\gamma}\,\Delta$
(2) The greedy policy's value is close to the estimate $V^\pi - V' = \big(\Tpi V^\pi - \Tpi V'\big) + \big(\Tstar V' - \Tstar V_k\big)$, $\;$ so $\;\norm{V^\pi - V'} \le \gamma \norm{V^\pi - V'} + \gamma \Delta$
$\Rightarrow\; \norm{V^\pi - V'} \le \dfrac{\gamma}{1-\gamma}\,\Delta$

Together: $\;\norm{V^\pi - \Vstar} \le \dfrac{2\gamma}{1-\gamma}\,\Delta < \varepsilon \;$ as soon as $\;\Delta < \varepsilon\dfrac{1-\gamma}{2\gamma}$. $\quad\blacksquare$

Bonus from (1): the value estimate itself is then within $\varepsilon/2$ of $\Vstar$.

References