Model-Free Prediction

Reinforcement Learning · Lecture 3
Claire Vernade
University of Technology Nuremberg
29 October 2026

Organisation

Today

  1. Recap: Bellman equations, value iteration and policy iteration
  2. From planning to learning: what if we don't know $P$ and $r$?
  3. Monte Carlo prediction
  4. Temporal-difference prediction and bootstrapping
  5. Bias vs. variance, and TD($\lambda$) in between
Today's question The policy $\pi$ is fixed. Can we estimate $\Vpi$ from experience alone?
Part 1

Recap: planning with a model

Same maze, same notation as last week.

RecapThe two-exit maze and its optimal values

$\Vstar$ and $\pistar$ for $\gamma = 0.9$

Deterministic moves. Exits: $+1$ (close), $+5$ (far). All other rewards $0$.

Bellman optimality equation $$\Vstar(s) = \max_a \Big( r(s,a) + \gamma \sum_{s'} P(s' \mid s,a)\, \Vstar(s') \Big)$$ $\pistar$ is greedy w.r.t. $\Vstar$.

At $\gamma = 0.9$ the far exit is worth it from everywhere: $V^\star(s_1) = 5\gamma^5 \approx 2.95$.

RecapValue iteration and policy iteration

Value iteration
  1. $V_0 = 0$
  2. Repeat, for all $s$: $$V_{k+1}(s) = \max_a \Big( \textcolor{#b5541c}{r(s,a)} + \gamma \sum_{s'} \textcolor{#b5541c}{P(s'|s,a)}\, V_k(s') \Big)$$
  3. Act greedily w.r.t. the last $V_k$
Policy iteration
  1. Start from any $\pi_0$
  2. Evaluate: $$V^{\pi_k} = \textcolor{#b5541c}{r^{\pi_k}} + \gamma\, \textcolor{#b5541c}{P^{\pi_k}}\, V^{\pi_k}$$
  3. Improve, for all $s$: $$\pi_{k+1}(s) = \argmax_a \big( \textcolor{#b5541c}{r(s,a)} + \gamma \textstyle\sum_{s'} \textcolor{#b5541c}{P(s'|s,a)}\, V^{\pi_k}(s') \big)$$
Both need the model Every update uses $\textcolor{#b5541c}{P}$ and $\textcolor{#b5541c}{r}$ explicitly. So far we did pure planning.

RecapVI and PI side by side

Value iteration: one sweep of $\Tstar$ per step
Policy iteration: evaluate, then improve

From planning to learning

Someone gives you a policy $\pi$: a direction to take in every cell (right).

But you know nothing about the world: not $P$, not $r$. You don't know where the exits are or what they pay.

All you can do is follow $\pi$ and observe: $$S_0, A_0, R_1, S_1, A_1, R_2, \dots, S_{T-1}, A_{T-1}, R_T$$

Discuss with your neighbour How can you find out how good this policy is, starting from each cell?
The policy you are given

Reinforcement learning
= generalised policy iteration from data

π V̂ evaluation: estimate V̂ ≈ Vπ from data improvement: π ← (more) greedy w.r.t. V̂ TODAY
  • Policy evaluation: estimate the current value function using data
  • Policy improvement: update the policy gradually towards the greedy one
  • Today: evaluation only (prediction). The policy is fixed.
  • Next week (Chris): improvement, i.e. control. SARSA and Q-learning.

Today's setting: a fixed policy

$\pi$ and its true value $\Vpi$ (hidden from the agent!)

$\pi$: with probability $0.8$ follow $\pistar$, with probability $0.2$ pick a random action.

  • Not optimal, not random: an arbitrary fixed policy
  • Episodes start in $s_1$, $\gamma = 0.9$
  • We know $\Vpi$ because we know the model, so we can check the agent's estimates
Goal (prediction) From episodes generated by $\pi$, estimate $\Vpi(s) = \E_\pi[G_t \mid S_t = s]$.
Part 2

Monte Carlo prediction

A value is an expectation, so estimate it by an average.

Monte Carlo: the idea

$$\Vpi(s) = \E_\pi\big[\,G_t \mid S_t = s\,\big] \quad\approx\quad \frac{1}{N(s)} \sum_{i=1}^{N(s)} G^{(i)}(s)$$
Requires Episodes that end: we need the whole return $G_t$ before we can update.
Does not need No model ($P$, $r$ unknown), no Bellman equation, not even the Markov property: we only average returns we observed.

Let's practice: returns of two episodes

$\gamma = 0.9$. Backwards: $\;G_t = R_{t+1} + \gamma\, G_{t+1}$, $\;G_T = 0$.

Episode 1 (follows π)
$t$$S_t$$R_{t+1}$$G_t$
0$s_1$02.95
1$s_2$03.28
2$s_3$03.65
3$s_6$04.05
4$s_7$04.50
5$s_{10}$55
Episode 2
$t$$S_t$$R_{t+1}$$G_t$
0$s_1$00.90
1$s_2$11

In $s_2$ the agent took one of its random actions ($\uparrow$) and fell into the small exit.

Estimate after 2 episodes: average the returns $V(s_1) = \tfrac{2.95 + 0.90}{2} = 1.93$, $\;V(s_2) = \tfrac{3.28 + 1}{2} = 2.14$; others: one return each.
States never visited keep their initial value $V = 0$: no data, no estimate.

Monte Carlo prediction: the algorithm

What if there are loops? A state can be visited several times in one episode: $\;\colorbox{#fbe3d2}{$s_1$} \to s_5 \to \colorbox{#fbe3d2}{$s_1$} \to s_2 \to s_3 \to \cdots \to +5$. Each visit of $s_1$ is followed by its own return. Which one(s) do we use?
Every-visit MC
  1. $V(s) = 0$, $N(s) = 0$ for all $s$
  2. For each episode $S_0, A_0, R_1, \dots, R_T$ generated by $\pi$:
    $G \leftarrow 0$; for $t = T-1, \dots, 0$:
    • $G \leftarrow R_{t+1} + \gamma G$
    • $N(S_t) \mathrel{+}= 1, \quad V(S_t) \leftarrow V(S_t) + \frac{1}{N(S_t)}\big(G - V(S_t)\big)$
Variant: first-visit MC Only update at the first visit of $S_t$ in the episode. The returns are then independent samples of $\Vpi(s)$, which makes the analysis simpler (the textbook version). Both converge to $\Vpi$.

Why $\tfrac{1}{N}$? From the average to a step size $\alpha$

After $N$ visits of $s$ with returns $G_1, \dots, G_N$, the Monte Carlo estimate is their average:

$$V_N = \frac{1}{N}\sum_{i=1}^{N} G_i = \frac{1}{N}\Big((N-1)\,V_{N-1} + G_N\Big) = V_{N-1} + \frac{1}{N}\big(G_N - V_{N-1}\big)$$
Constant step size Replace $\tfrac1N$ by a constant $\textcolor{#b5541c}{\alpha} \in (0,1)$:
$$V \leftarrow V + \textcolor{#b5541c}{\alpha}\,\big(G - V\big) \quad\Longrightarrow\quad V_N = (1-\textcolor{#b5541c}{\alpha})^N V_0 + \textstyle\sum_{i=1}^{N} \textcolor{#b5541c}{\alpha}(1-\textcolor{#b5541c}{\alpha})^{N-i}\, G_i$$

Recent returns count more; old ones are forgotten geometrically. Useful when the policy keeps changing (next week).

Let's practice: MC by hand, then many episodes

First the three episodes below. Step plays one transition; at the end of an episode, each Step performs one update, going backwards.

  1. Ep. 1: $s_1 \to s_2 \to s_3 \to s_6 \to s_7 \to s_{10} \to$ +5
  2. Ep. 2: $s_1 \to s_2 \to$ +1  (random action in $s_2$)
  3. Ep. 3: $s_1 \to s_5 \to s_1 \to s_2 \to \cdots \to$ +5  (revisits $s_1$)

Then +10, +100: new random episodes from $\pi$. Compare with the true $\Vpi$. ($n$ = number of updates of each cell.)

Discuss Which cells are learned fast, which barely at all, and why? What would go wrong if an episode never ended?

Monte Carlo: properties

Good

  • Unbiased: $\E[G_t \mid S_t = s] = \Vpi(s)$
  • Simple; no model needed
  • Does not rely on the Markov property

Less good

  • High variance: $G_t$ sums many random rewards and transitions
  • Must wait until the end of the episode to update
  • Only for episodic tasks
Can we do better? Bellman: $\Vpi(s) = \E_\pi[R_{t+1} + \gamma\, \Vpi(S_{t+1}) \mid S_t = s]$. Could we use that, without $P$?

Questions?

Part 3

Temporal-difference learning

Learn a guess from a guess.

TD(0): the idea

By definition of the return: $\quad G_t = R_{t+1} + \gamma\, \cfut{G_{t+1}}$

MC uses $G_t$ as the target, so it must wait until the episode ends to know $\cfut{G_{t+1}}$.

TD: after one step we don't know $\cfut{G_{t+1}}$ yet, but $\cfut{V(S_{t+1})}$ is our current estimate of it, since $\Vpi(s') = \E_\pi[G_{t+1} \mid S_{t+1} = s']$:

$$V(S_t) \leftarrow V(S_t) + \alpha \big( \underbrace{R_{t+1} + \gamma\, \underbrace{\cfut{V(S_{t+1})}}_{\text{estimates } G_{t+1}}}_{\text{TD target: estimates } G_t} - V(S_t) \big)$$
TD error $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$
with $V(\text{exit}) = 0$
  • Update after every step, online
  • Works for continuing tasks (no end needed)
  • A sample of the Bellman expectation equation
  • We say the estimator bootstraps: it uses its own stored values $V(S_{t+1})$

Backup diagrams: what each target looks at

DP

$V(s) \leftarrow \sum_a \cpol{\pi(a|s)} \big( \crew{r(s,a)}$
$\qquad\qquad + \gamma \sum_{s'} \cfut{P(s'|s,a)\, V(s')} \big)$

s π(a|s) P(s′|s,a) bootstrap: V(s′)

Needs the model. One step, all branches.

MC

$V(S_t) \leftarrow V(S_t) + \alpha\big(\crew{G_t} - V(S_t)\big)$, $\;\crew{G_t = R_{t+1} + \gamma R_{t+2} + \cdots}$

St At Rt+1, St+1 At+1 Rt+2, St+2 ⋮ RT, endGt

One sampled trajectory, all the way to the end.

TD(0)

$V(S_t) \leftarrow V(S_t) + \alpha\big(\crew{R_{t+1}} + \gamma\, \cfut{V(S_{t+1})} - V(S_t)\big)$

St At Rt+1, St+1bootstrap: V(St+1)rest of the episode:not needed

One sampled step, then bootstrap.

Let's practice: TD(0) by hand, same episodes

$\alpha = 0.5$, $V_0 = 0$. Each Step = one transition and its update.

  • After episode 1, which states have changed? Only $s_{10}$: $V(s_{10}) = 2.5$
  • After episode 3: $V(s_7) = 1.125$, $V(s_{10}) = 3.75$. Information travels one step back per visit.

MazeMC and TD on the same episodes

Discuss After 10 episodes, why is TD's $V(s_1)$ still close to $0$? What do you expect after 100?

The $+5$ moves back one state per visit, scaled by $\alpha$. MC moves it all the way back in one episode.

Here the conditions are ideal for MC: short episodes, a single reward at the end, deterministic moves. See the appendix for the bigger maze.

Part 4

Bias, variance, and TD($\lambda$)

MazeBias and variance of the estimate of $V(s_2)$

Exactly the setting of the previous slide: same maze and policy, $\gamma = 0.9$, start in $s_1$, $V_0 = 0$, both methods with $\alpha = 0.2$. We repeat the 100-episode run 1000 times and record $V(s_2)$ after every episode.

Bias TD starts far too low and needs about 50 episodes to catch up: it bootstraps from $V(s_3), V(s_6), \dots$, which are still $0$. MC only has to forget $V_0$ (about 10 episodes).
Variance Once settled, MC's estimates spread more (sd ≈ 0.29 vs ≈ 0.20): the MC target adds up all the random steps until the exit, the TD target only one.

Bias–variance trade-off

TargetBiasVariance
MC$G_t = R_{t+1} + \gamma R_{t+2} + \dots$none: $\E[G_t | S_t = s] = \Vpi(s)$high: many random steps
TD$R_{t+1} + \gamma V(S_{t+1})$biased while $V \neq \Vpi$low: one random step
Next Is there a middle ground that trades off bias and variance?

A detailed bias–variance study in the bigger maze is in the appendix.

In between: $n$-step returns

Look $n$ steps ahead, then bootstrap:

$$G_t^{(n)} = R_{t+1} + \gamma R_{t+2} + \dots + \gamma^{n-1} R_{t+n} + \gamma^n V(S_{t+n})$$
$n$$1$$2$$\cdots$$\infty$
target$R_{t+1} + \gamma V(S_{t+1})$  (TD(0))$R_{t+1} + \gamma R_{t+2} + \gamma^2 V(S_{t+2})$$G_t$  (MC)

$n$-step TD: $\;V(S_t) \leftarrow V(S_t) + \alpha\big(G_t^{(n)} - V(S_t)\big)$. Small $n$: more bias. Large $n$: more variance.

DiscussWhich $n$ should we choose? Do we have to choose just one?

TD($\lambda$): average over all $n$

$\lambda$-return $$G_t^\lambda = (1-\lambda) \sum_{n=1}^{\infty} \lambda^{n-1} G_t^{(n)}, \qquad \lambda \in [0,1]$$
  • Geometric weights, summing to 1
  • $\lambda = 0$: TD(0). $\;\lambda = 1$: MC.
  • "Forward view": still needs the future, like MC
1234567… n weight (1−λ)λⁿ⁻¹, λ = 0.7

TD($\lambda$), backward view: eligibility traces

TD($\lambda$) (accumulating traces)
  1. Each episode: $e(s) = 0$ for all $s$
  2. At each step $t$:
    • $\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t)$
    • $e(s) \leftarrow \gamma \lambda\, e(s)$ for all $s$; $\quad e(S_t) \leftarrow e(S_t) + 1$
    • $V(s) \leftarrow V(s) + \alpha\, \delta_t\, e(s)$ for all $s$
  • $e(s)$ is a discounted counter of visits: $+1$ at each visit of $s$, then shrinks by $\gamma\lambda$ at every step: $\;e_t(s) = \sum_{k \le t} (\gamma\lambda)^{t-k}\, \mathbb{1}[S_k = s]$
  • Online, one step at a time, like TD(0)
  • Sends each TD error back to all recently visited states at once
  • Offline, it matches the forward view (Sutton & Barto §12.4)

Example: episode $s_1 \to s_5 \to s_1 \to s_2 \to s_3$, with $\gamma\lambda = 0.5$. At $t = 4$, the TD error $\delta_4$ updates every visited state, weighted by its trace (box on the right).

0.5 1 0 trace e(s)time tvisited state 0 s1 1 s5 2 s1 3 s2 4 s3 1 0.5 0.625 1 0.5 0.25 1 1.25revisit: 0.25 + 1 weights on δ₄ at t = 4 e(s3) = 1 e(s2) = 0.5 e(s1) = 0.3125 e(s5) = 0.125

MazeTD($\lambda$) on the same episodes

Discuss Play one episode. Which states did TD(0) update? Which did TD($\lambda$) update?

TD($\lambda$) sends the final TD error back along the whole path, with weight $(\gamma\lambda)^k$.

MazeChoosing $\lambda$

Average error over the first 30 episodes, for four step sizes $\alpha$.

  • Small $\alpha$: $\lambda$ close to 1 is best. The bias from $V_0 = 0$ dominates, so it pays to look far ahead.
  • Larger $\alpha$: the best $\lambda$ is in between (about 0.6–0.8). Variance matters too.
  • Same picture as Sutton & Barto, Fig. 12.3

Is TD($\lambda$) still used?

From $V$ to $Q$: $n$-step TD for action values

Same ideas, with state–action pairs instead of states: estimate $\Qpi(s,a) = \E_\pi[G_t \mid S_t = s, A_t = a]$, a table of $|\S| \times |\A|$ numbers (48 in the maze).

Along the episode $\;S_t, A_t, \crew{R_{t+1}}, S_{t+1}, A_{t+1}, \crew{R_{t+2}}, \dots, S_{t+n}, A_{t+n}$, with all actions drawn from $\pi$:

$$\begin{aligned} G_{t:t+n} &= \crew{R_{t+1} + \gamma R_{t+2} + \dots + \gamma^{n-1} R_{t+n}} + \gamma^n \cfut{Q(S_{t+n}, A_{t+n})} \\ Q(S_t, A_t) &\leftarrow Q(S_t, A_t) + \alpha \big( G_{t:t+n} - Q(S_t, A_t) \big) \end{aligned}$$
$n = 1$: TD(0) for $Q$ $Q(S_t,A_t) \leftarrow Q(S_t,A_t) + \alpha\big(\crew{R_{t+1}} + \gamma\, \cfut{Q(S_{t+1}, A_{t+1})} - Q(S_t,A_t)\big)$
Uses the quintuple $(S_t, A_t, R_{t+1}, S_{t+1}, A_{t+1})$.
$n = \infty$: Monte Carlo for $Q$ Average the returns $G_t$ observed after each visit of the pair $(s,a)$. Same $\lambda$-trick as for $V$ in between.
Why bother with $Q$? Without a model, $V$ alone cannot tell us which action is best: $\argmax_a \big(r(s,a) + \gamma \sum_{s'} P(s'|s,a) V(s')\big)$ needs $P$ and $r$. With $Q$, the greedy action is simply $\argmax_a Q(s,a)$.

ConclusionsTwo ways to estimate a value function from data

Monte Carlo Target: the full return $\crew{G_t}$
  • Unbiased, but high variance
  • Waits for the end of the episode
  • Needs no Markov property
Temporal difference Target: $\crew{R_{t+1}} + \gamma\, \cfut{V(S_{t+1})}$ (bootstrapping)
  • Biased while $V$ is wrong, but low variance
  • Learns after every step
  • Uses the Markov structure

In between: $n$-step returns and TD($\lambda$) choose how far to look before bootstrapping. The same ideas estimate $Q$ as well as $V$.

Why TD remains the workhorse of RL

The TD error is the gap between the current estimate and the (sampled) Bellman target:

$$\delta_t = \underbrace{\crew{R_{t+1}} + \gamma\, \cfut{V(S_{t+1})}}_{\text{Bellman target}} - \underbrace{V(S_t)}_{\text{estimate}}$$

It works like a loss: $V(S_t) \leftarrow V(S_t) + \alpha\, \delta_t$ is a gradient step on $\tfrac12 \delta_t^2$ (target held fixed). Learning a value function becomes "make the Bellman error small", for any model of $V$ or $Q$.

Next time: Model-free control (Chris)

ThinkWhat changes if we bootstrap from $\max_a Q(S_{t+1}, a)$ instead of $Q(S_{t+1}, A_{t+1})$?
Appendix

Bias–variance study

MC and TD compared in the bigger maze: spread of the targets, online learning, and what each method makes of the same data.

AppendixA bigger maze, with rewards along the way

The given policy $\pi$ and its true value $\Vpi$ (hover a cell for the number)

The two-exit maze is small, with a single reward at the end. What about bigger problems with rewards along the way? A 10×10 maze:

  • Coins $+1$ and traps $-1$, each time you step on them
  • Exits: $+5$ (far, behind a trap), $+1$ (near the start)
  • Slippery: 20% chance to slip
  • Given policy: the shortest route to the big exit, with 20% random actions
  • $\gamma = 0.95$; episodes last about 30 steps

Every return $G_t$ now adds up many random rewards: coins and traps hit or missed, slips.

Bigger mazeMonte Carlo in the bigger maze

  • Press Episode a few times, then +10, +100
  • Each episode updates every cell along its path, with a very noisy return
  • Cells off the route are rarely visited, so they stay wrong for a long time
Discuss Two episodes pass through the same cell and then continue the same way. Does MC "notice" that they share the rest of the path?

Bigger mazeWhat do the targets for $V(\text{start})$ look like?

300 episodes from the start cell. Each row is a histogram: bar height = how many of the 300 targets fall in that range. Black line = their mean.

Bigger mazeMC and TD online, on the same episodes

Discuss Run +100 a few times. Where is TD better already? Where is it still behind?

Near the exits TD is quickly accurate. Far from them, the $+5$ still has to travel back one cell per visit.

Bigger mazeLearning curves, online

Average over 20 runs; error weighted by how often $\pi$ visits each state.

  • Online, from $V_0 = 0$, MC is still ahead early on: TD pays for its bias while values propagate back
  • A larger $\alpha$ speeds TD up but leaves more noise at the end
  • So is TD just worse? Look at what happens when both get the same data and can use it fully.

Bigger mazeSame data, better answer: batch TD

Batch setting: collect $K$ episodes, then replay them again and again until the estimates stop changing.

  • Batch MC = average of the observed returns
  • Batch TD(0) = exact value of the MDP estimated from the data (empirical transitions and rewards)
  • TD is about 25% more accurate for every amount of data: it uses the Markov structure, MC does not

Bigger mazeWhy: TD reuses what it knows about the future

Dashed: a new start cell, one step off the usual route

50 ordinary episodes from the start, then one episode from the new cell. How good is each estimate of $V(\text{new cell})$?

  • MC: the only data is that one noisy return
  • TD: one step, then it joins states it already knows well. It bootstraps from them.

References