给定这么一个例子和策略
Let v i v_i v i s i s_i s i i = 1 , 2 , 3 , 4 i = 1, 2, 3, 4 i = 1 , 2 , 3 , 4
v 1 = r 1 + γ r 2 + γ 2 r 3 + … v 2 = r 2 + γ r 3 + γ 2 r 4 + … v 3 = r 3 + γ r 4 + γ 2 r 1 + … v 4 = r 4 + γ r 1 + γ 2 r 2 + … \begin{aligned}
v_1 &= r_1 + \gamma r_2 + \gamma^2 r_3 + \dots \\
v_2 &= r_2 + \gamma r_3 + \gamma^2 r_4 + \dots \\
v_3 &= r_3 + \gamma r_4 + \gamma^2 r_1 + \dots \\
v_4 &= r_4 + \gamma r_1 + \gamma^2 r_2 + \dots
\end{aligned} v 1 v 2 v 3 v 4 = r 1 + γ r 2 + γ 2 r 3 + … = r 2 + γ r 3 + γ 2 r 4 + … = r 3 + γ r 4 + γ 2 r 1 + … = r 4 + γ r 1 + γ 2 r 2 + … v 1 = r 1 + γ ( r 2 + γ r 3 + … ) = r 1 + γ v 2 v 2 = r 2 + γ ( r 3 + γ r 4 + … ) = r 2 + γ v 3 v 3 = r 3 + γ ( r 4 + γ r 1 + … ) = r 3 + γ v 4 v 4 = r 4 + γ ( r 1 + γ r 2 + … ) = r 4 + γ v 1 \begin{aligned}
v_1 &= r_1 + \gamma(r_2 + \gamma r_3 + \dots) = r_1 + \gamma v_2 \\
v_2 &= r_2 + \gamma(r_3 + \gamma r_4 + \dots) = r_2 + \gamma v_3 \\
v_3 &= r_3 + \gamma(r_4 + \gamma r_1 + \dots) = r_3 + \gamma v_4 \\
v_4 &= r_4 + \gamma(r_1 + \gamma r_2 + \dots) = r_4 + \gamma v_1
\end{aligned} v 1 v 2 v 3 v 4 = r 1 + γ ( r 2 + γ r 3 + … ) = r 1 + γ v 2 = r 2 + γ ( r 3 + γ r 4 + … ) = r 2 + γ v 3 = r 3 + γ ( r 4 + γ r 1 + … ) = r 3 + γ v 4 = r 4 + γ ( r 1 + γ r 2 + … ) = r 4 + γ v 1
The returns rely on each other. Bootstrapping! 相互依赖的。
How to solve these equations? Write in the following matrix-vector form:
[ v 1 v 2 v 3 v 4 ] ⏟ v = [ r 1 r 2 r 3 r 4 ] + [ γ v 2 γ v 3 γ v 4 γ v 1 ] = [ r 1 r 2 r 3 r 4 ] ⏟ r + γ [ 0 1 0 0 0 0 1 0 0 0 0 1 1 0 0 0 ] ⏟ P [ v 1 v 2 v 3 v 4 ] ⏟ v \underbrace{
\begin{bmatrix} v_1 \\ v_2 \\ v_3 \\ v_4 \end{bmatrix}
}_{\mathbf{v}}
=
\begin{bmatrix} r_1 \\ r_2 \\ r_3 \\ r_4 \end{bmatrix}
+
\begin{bmatrix} \gamma v_2 \\ \gamma v_3 \\ \gamma v_4 \\ \gamma v_1 \end{bmatrix}
=
\underbrace{
\begin{bmatrix} r_1 \\ r_2 \\ r_3 \\ r_4 \end{bmatrix}
}_{\mathbf{r}}
+ \gamma
\underbrace{
\begin{bmatrix}
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
1 & 0 & 0 & 0
\end{bmatrix}
}_{\mathbf{P}}
\underbrace{
\begin{bmatrix} v_1 \\ v_2 \\ v_3 \\ v_4 \end{bmatrix}
}_{\mathbf{v}} v v 1 v 2 v 3 v 4 = r 1 r 2 r 3 r 4 + γ v 2 γ v 3 γ v 4 γ v 1 = r r 1 r 2 r 3 r 4 + γ P 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 v v 1 v 2 v 3 v 4 which can be rewritten as
v = r + γ P v \mathbf{v} = \mathbf{r} + \gamma \mathbf{P} \mathbf{v} v = r + γ Pv 此处P \mathbf P P
接下去移项求逆：
因此：
v = ( I − γ P ) − 1 r \boxed{
\mathbf v
=
(I-\gamma P)^{-1}\mathbf r
} v = ( I − γ P ) − 1 r 这就是直接求解 state value 的方法。
Consider the following single-step process:
S t → A t R t + 1 , S t + 1 S_t \xrightarrow{A_t} R_{t+1}, S_{t+1} S t A t R t + 1 , S t + 1
t , t + 1 t, t + 1 t , t + 1 S t S_t S t t t t A t A_t A t S t S_t S t R t + 1 R_{t+1} R t + 1 A t A_t A t R t R_t R t S t + 1 S_{t+1} S t + 1 A t A_t A t
Note that S t , A t , R t + 1 S_t, A_t, R_{t+1} S t , A t , R t + 1 random variables . 随机变量，意思就是可以进行求期望等操作
This step is governed by （由…决定）the following probability distributions:
S t → A t S_t \to A_t S t → A t π ( A t = a ∣ S t = s ) \pi(A_t = a \mid S_t = s) π ( A t = a ∣ S t = s ) S t , A t → R t + 1 S_t, A_t \to R_{t+1} S t , A t → R t + 1 p ( R t + 1 = r ∣ S t = s , A t = a ) p(R_{t+1} = r \mid S_t = s, A_t = a) p ( R t + 1 = r ∣ S t = s , A t = a ) S t , A t → S t + 1 S_t, A_t \to S_{t+1} S t , A t → S t + 1 p ( S t + 1 = s ′ ∣ S t = s , A t = a ) p(S_{t+1} = s' \mid S_t = s, A_t = a) p ( S t + 1 = s ′ ∣ S t = s , A t = a )
At this moment, we assume we know the model (i.e., the probability distributions)!
Consider the following multi-step trajectory:
S t → A t R t + 1 , S t + 1 → A t + 1 R t + 2 , S t + 2 → A t + 2 R t + 3 , … S_t \xrightarrow{A_t} R_{t+1}, S_{t+1} \xrightarrow{A_{t+1}} R_{t+2}, S_{t+2} \xrightarrow{A_{t+2}} R_{t+3}, \dots S t A t R t + 1 , S t + 1 A t + 1 R t + 2 , S t + 2 A t + 2 R t + 3 , … The discounted return is
G t = R t + 1 + γ R t + 2 + γ 2 R t + 3 + … G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots G t = R t + 1 + γ R t + 2 + γ 2 R t + 3 + …
γ ∈ ( 0 , 1 ) \gamma \in (0, 1) γ ∈ ( 0 , 1 ) G t G_t G t R t + 1 , R t + 2 , … R_{t+1}, R_{t+2}, \dots R t + 1 , R t + 2 , …
G t G_t G t
v π ( s ) = E [ G t ∣ S t = s ] v_{\pi}(s) = \mathbb E [G_t \mid S_t = s] v π ( s ) = E [ G t ∣ S t = s ] 它由s , π s,\pi s , π v ( π , s ) v(\pi,s) v ( π , s )
Q: What is the relationship between return and state value?
A: The state value is the mean of all possible returns that can be obtained starting from a state. If everything - π ( a ∣ s ) , p ( r ∣ s , a ) , p ( s ′ ∣ s , a ) \pi(a|s), p(r|s, a), p(s'|s, a) π ( a ∣ s ) , p ( r ∣ s , a ) , p ( s ′ ∣ s , a )
贝尔曼公式描述了不同state value之间的关系，接下去，我们来推导（derive）贝尔曼公式
还是考虑这么一个轨迹：
S t → A t R t + 1 , S t + 1 → A t + 1 R t + 2 , S t + 2 → A t + 2 R t + 3 , … S_t \xrightarrow{A_t} R_{t+1}, S_{t+1} \xrightarrow{A_{t+1}} R_{t+2}, S_{t+2} \xrightarrow{A_{t+2}} R_{t+3}, \dots S t A t R t + 1 , S t + 1 A t + 1 R t + 2 , S t + 2 A t + 2 R t + 3 , … 根据discounted return的定义，我们可以将G t G_t G t
G t = R t + 1 + γ R t + 2 + γ 2 R t + 3 + … = R t + 1 + γ ( R t + 2 + γ R t + 3 + … ) = R t + 1 + γ G t + 1 \begin{aligned}
G_t &= R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \dots \\
& = R_{t+1} + \gamma (R_{t+2} + \gamma R_{t+3} + \dots) \\
&= R_{t+1} + \gamma G_{t+1}
\end{aligned}
G t = R t + 1 + γ R t + 2 + γ 2 R t + 3 + … = R t + 1 + γ ( R t + 2 + γ R t + 3 + … ) = R t + 1 + γ G t + 1 根据state value的定义：
v π ( s ) = E [ G t ∣ S t = s ] = E [ R t + 1 + γ G t + 1 ∣ S t = s ] = E [ R t + 1 ∣ S t = s ] + γ E [ G t + 1 ∣ S t = s ] v_{\pi}(s) = \mathbb E[G_t \mid S_t = s] = \mathbb E[R_{t+1} + \gamma G_{t+1} \mid S_t = s] \\
= \mathbb E[R_{t+1} \mid S_t = s] + \gamma \mathbb E[ G_{t+1} \mid S_t = s] v π ( s ) = E [ G t ∣ S t = s ] = E [ R t + 1 + γ G t + 1 ∣ S t = s ] = E [ R t + 1 ∣ S t = s ] + γ E [ G t + 1 ∣ S t = s ] 接下来分别计算这两个期望，
我们知道R t + 1 R_{t+1} R t + 1 S t + 1 S_{t+1} S t + 1 S t S_t S t A t A_t A t
E [ R t + 1 ∣ S t = s ] = ∑ a π ( a ∣ s ) E [ R t + 1 ∣ S t = s , A t = a ] = ∑ a π ( a ∣ s ) ∑ r r p ( r ∣ s , a ) \mathbb E[R_{t+1} \mid S_t = s] = \sum_{a} \pi(a|s) \mathbb E[R_{t+1} \mid S_t = s, A_t = a] \\
= \sum_{a} \pi(a|s) \sum_{r} r p(r | s, a) E [ R t + 1 ∣ S t = s ] = a ∑ π ( a ∣ s ) E [ R t + 1 ∣ S t = s , A t = a ] = a ∑ π ( a ∣ s ) r ∑ r p ( r ∣ s , a ) This is the mean of immediate rewards
计算E [ G t + 1 ∣ S t = s ] \mathbb E[G_{t+1} \mid S_t = s] E [ G t + 1 ∣ S t = s ]
E [ G t + 1 ∣ S t = s ] = ∑ s ′ E [ G t + 1 ∣ S t = s , S t + 1 = s ′ ] p ( s ′ ∣ s ) 全期望公式，设下一步状态 s ′ = ∑ s ′ E [ G t + 1 ∣ S t + 1 = s ′ ] p ( s ′ ∣ s ) Markov process is memoryless = ∑ s ′ v π ( s ′ ) p ( s ′ ∣ s ) definition of state value = ∑ s ′ v π ( s ′ ) ∑ a p ( s ′ ∣ s , a ) π ( a ∣ s ) \begin{aligned}
\mathbb E[G_{t+1} \mid S_t = s] &= \sum_{s'} \mathbb E[G_{t+1} \mid S_t=s, S_{t+1} = s']p(s' \mid s) \quad 全期望公式，设下一步状态s' \\
&= \sum_{s'} \mathbb E[G_{t+1} \mid S_{t+1} = s'] p(s' \mid s) \quad \text{Markov process is memoryless }\\
&= \sum_{s'} v_{\pi}(s') p(s' \mid s) \quad \text{definition of state value }\\
&= \sum_{s'} v_{\pi}(s') \sum_{a} p(s' \mid s,a) \pi(a \mid s) \\
\end{aligned} E [ G t + 1 ∣ S t = s ] = s ′ ∑ E [ G t + 1 ∣ S t = s , S t + 1 = s ′ ] p ( s ′ ∣ s ) 全期望公式，设下一步状态 s ′ = s ′ ∑ E [ G t + 1 ∣ S t + 1 = s ′ ] p ( s ′ ∣ s ) Markov process is memoryless = s ′ ∑ v π ( s ′ ) p ( s ′ ∣ s ) definition of state value = s ′ ∑ v π ( s ′ ) a ∑ p ( s ′ ∣ s , a ) π ( a ∣ s ) This is the mean of future rewards
p(s’|s,a)
在一般的 MDP 里，即使已经确定了当前 state s s s a a a s ′ s' s ′ p ( s ′ ∣ s , a ) p(s'\mid s,a) p ( s ′ ∣ s , a )
比如你现在处于 s s s a a a
p ( s 1 ′ ∣ s , a ) = 0.8 , p ( s 2 ′ ∣ s , a ) = 0.2 p(s_1'\mid s,a)=0.8,\qquad p(s_2'\mid s,a)=0.2 p ( s 1 ′ ∣ s , a ) = 0.8 , p ( s 2 ′ ∣ s , a ) = 0.2 也就是：我已经确定要向右了，但环境可能有 20% 的概率让我滑到另一个位置
Therefore, we have
v π ( s ) = E [ R t + 1 ∣ S t = s ] + γ E [ G t + 1 ∣ S t = s ] , = ∑ a π ( a ∣ s ) ∑ r p ( r ∣ s , a ) r ⏟ mean of immediate rewards + γ ∑ a π ( a ∣ s ) ∑ s ′ p ( s ′ ∣ s , a ) v π ( s ′ ) ⏟ mean of future rewards , = ∑ a π ( a ∣ s ) [ ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v π ( s ′ ) ] , ∀ s ∈ S . \begin{aligned}
{\color{red}v_{\pi}(s)} &= \mathbb{E}[R_{t+1}|S_t = s] + \gamma \mathbb{E}[G_{t+1}|S_t = s], \\
&= {\color{blue}\underbrace{ {\color{blue}\sum_{a}\pi(a|s)\sum_{r}p(r|s,a)}{\color{black}r} }_{\text{mean of immediate rewards}}} + \gamma {\color{blue}\underbrace{ {\color{blue}\sum_{a}\pi(a|s)\sum_{s'}p(s'|s,a)}{\color{red}v_{\pi}(s')} }_{\text{mean of future rewards}}}, \\
&= {\color{blue}\sum_{a}\pi(a|s)} {\color{blue}\left[ {\color{blue}\sum_{r}p(r|s,a)}{\color{black}r} + \gamma {\color{blue}\sum_{s'}p(s'|s,a)}{\color{red}v_{\pi}(s')} \right]}, \quad \forall s \in \mathcal{S}.
\end{aligned} v π ( s ) = E [ R t + 1 ∣ S t = s ] + γ E [ G t + 1 ∣ S t = s ] , = mean of immediate rewards a ∑ π ( a ∣ s ) r ∑ p ( r ∣ s , a ) r + γ mean of future rewards a ∑ π ( a ∣ s ) s ′ ∑ p ( s ′ ∣ s , a ) v π ( s ′ ) , = a ∑ π ( a ∣ s ) [ r ∑ p ( r ∣ s , a ) r + γ s ′ ∑ p ( s ′ ∣ s , a ) v π ( s ′ ) ] , ∀ s ∈ S . Highlights:
The above equation is called the Bellman equation , which characterizes the relationship among the state-value functions of different states.
It consists of two terms: the immediate reward term and the future reward term .
A set of equations: every state has an equation like this!!!
可以先把里面的变量分成三组：
v π ( s ) , v π ( s ′ ) ⏟ 要求的 , π ( a ∣ s ) ⏟ 给定的策略 , p ( r ∣ s , a ) , p ( s ′ ∣ s , a ) ⏟ 环境模型 \underbrace{v_\pi(s),v_\pi(s')}_{\text{要求的}}
,\qquad
\underbrace{\pi(a|s)}_{\text{给定的策略}}
,\qquad
\underbrace{p(r|s,a),p(s'|s,a)}_{\text{环境模型}} 要求的 v π ( s ) , v π ( s ′ ) , 给定的策略 π ( a ∣ s ) , 环境模型 p ( r ∣ s , a ) , p ( s ′ ∣ s , a )
v π ( s ) v_\pi(s) v π ( s ) v π ( s ′ ) v_\pi(s') v π ( s ′ ) 这里π \pi π π \pi π v π v_{\pi} v π
p ( r ∣ s , a ) p(r|s,a) p ( r ∣ s , a ) p ( s ′ ∣ s , a ) p(s'|s,a) p ( s ′ ∣ s , a )
reward model p ( r ∣ s , a ) p(r \mid s, a) p ( r ∣ s , a )
transition model p ( s ′ ∣ s , a ) p(s' \mid s, a) p ( s ′ ∣ s , a )
上一节得出的贝尔曼公式是element-wise的，它对于每一个state s ∈ S s \in \mathcal S s ∈ S ∣ S ∣ |\mathcal S| ∣ S ∣
Recall that:
v π ( s ) = ∑ a π ( a ∣ s ) [ ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v π ( s ′ ) ] v_{\pi}(s) = \sum_{a} \pi(a|s) \left[ \sum_{r} p(r|s, a) r + \gamma \sum_{s'} p(s'|s, a) v_{\pi}(s') \right] v π ( s ) = a ∑ π ( a ∣ s ) [ r ∑ p ( r ∣ s , a ) r + γ s ′ ∑ p ( s ′ ∣ s , a ) v π ( s ′ ) ] Rewrite the Bellman equation as
v π ( s ) = r π ( s ) + γ ∑ s ′ p π ( s ′ ∣ s ) v π ( s ′ ) (1) v_{\pi}(s) = r_{\pi}(s) + \gamma \sum_{s'} p_{\pi}(s'|s) v_{\pi}(s') \tag{1} v π ( s ) = r π ( s ) + γ s ′ ∑ p π ( s ′ ∣ s ) v π ( s ′ ) ( 1 ) where
r π ( s ) ≜ ∑ a π ( a ∣ s ) ∑ r p ( r ∣ s , a ) r , p π ( s ′ ∣ s ) ≜ ∑ a π ( a ∣ s ) p ( s ′ ∣ s , a ) r_{\pi}(s) \triangleq \sum_{a} \pi(a|s) \sum_{r} p(r|s, a) r, \qquad p_{\pi}(s'|s) \triangleq \sum_{a} \pi(a|s) p(s'|s, a) r π ( s ) ≜ a ∑ π ( a ∣ s ) r ∑ p ( r ∣ s , a ) r , p π ( s ′ ∣ s ) ≜ a ∑ π ( a ∣ s ) p ( s ′ ∣ s , a ) 其实r π ( s ) r_{\pi}(s) r π ( s ) E [ R t + 1 ∣ S t = s ] \mathbb E[R_{t+1} \mid S_t = s] E [ R t + 1 ∣ S t = s ] p π ( s ′ ∣ s ) p_{\pi}(s'|s) p π ( s ′ ∣ s ) s s s s ′ s' s ′
设状态s 1 , s 2 , . . . s n s_1,s_2,...s_n s 1 , s 2 , ... s n s i s_i s i
v π ( s i ) = r π ( s i ) + γ ∑ s j p π ( s j ∣ s i ) v π ( s j ) v_{\pi}(s_i) = r_{\pi}(s_i) + \gamma \sum_{s_j} p_{\pi}(s_j \mid s_i) v_{\pi}(s_j) v π ( s i ) = r π ( s i ) + γ s j ∑ p π ( s j ∣ s i ) v π ( s j ) 写成矩阵形式：
v π = r π + γ P π v π v_{\pi} = r_{\pi} + \gamma P_{\pi} v_{\pi} v π = r π + γ P π v π where
v π = [ v π ( s 1 ) , … , v π ( s n ) ] T ∈ R n v_{\pi} = [v_{\pi}(s_1), \dots, v_{\pi}(s_n)]^T \in \mathbb{R}^n v π = [ v π ( s 1 ) , … , v π ( s n ) ] T ∈ R n r π = [ r π ( s 1 ) , … , r π ( s n ) ] T ∈ R n r_{\pi} = [r_{\pi}(s_1), \dots, r_{\pi}(s_n)]^T \in \mathbb{R}^n r π = [ r π ( s 1 ) , … , r π ( s n ) ] T ∈ R n P π ∈ R n × n P_{\pi} \in \mathbb{R}^{n \times n} P π ∈ R n × n [ P π ] i j = p π ( s j ∣ s i ) [P_{\pi}]_{ij} = p_{\pi}(s_j|s_i) [ P π ] ij = p π ( s j ∣ s i ) state transition matrix
这个[ P π ] i j = p π ( s j ∣ s i ) [P_{\pi}]_{ij} = p_{\pi}(s_j|s_i) [ P π ] ij = p π ( s j ∣ s i ) s i s_i s i s j s_j s j
举个例子：
Given a policy, finding out the corresponding state values is called policy
evaluation!
贝尔曼方程：
v π = r π + γ P π v π v_{\pi} = r_{\pi} + \gamma P_{\pi} v_{\pi} v π = r π + γ P π v π 移项求逆：
v π = ( I − γ P π ) − 1 r π v_{\pi} = (I - \gamma P_{\pi})^{-1} r_{\pi} v π = ( I − γ P π ) − 1 r π 但是矩阵求逆可能很贵, 下面给出迭代法：
定义
v k = 第 k 次迭代时，对真实 v π 的估计 \mathbf v_k = \text{第 }k\text{ 次迭代时，对真实 }\mathbf v_\pi\text{ 的估计} v k = 第 k 次迭代时，对真实 v π 的估计 令
v k + 1 = r π + γ P π v k v_{k+1} = r_{\pi} + \gamma P_{\pi} v_{k} v k + 1 = r π + γ P π v k 先随便猜一个 value vector v 0 v_0 v 0
lim k → ∞ v k = v π = ( I − γ P π ) − 1 r π \lim_{k\rightarrow\infty} v_k = v_{\pi} = (I - \gamma P_{\pi})^{-1} r_{\pi} k → ∞ lim v k = v π = ( I − γ P π ) − 1 r π From state value to action value:
State value: the average return the agent can get starting from a state.
Action value: the average return the agent can get starting from a state and
taking an action
定义Action Value：
q π ( s , a ) = E [ G t ∣ S t = s , A t = a ] q_{\pi} (s,a) = \mathbb E[G_t \mid S_t=s, A_t=a] q π ( s , a ) = E [ G t ∣ S t = s , A t = a ]
q π ( s , a ) q_{\pi} (s,a) q π ( s , a ) ( s , a ) (s,a) ( s , a ) π \pi π
由全期望公式：
E [ G t ∣ S t = s ] ⏟ v π ( s ) = ∑ a E [ G t ∣ S t = s , A t = a ] ⏟ q π ( s , a ) π ( a ∣ s ) \underbrace{\mathbb{E}[G_t \mid S_t = s]}_{v_{\pi}(s)} = \sum_{a} \underbrace{\mathbb{E}[G_t \mid S_t = s, A_t = a]}_{q_{\pi}(s,a)} \pi(a|s) v π ( s ) E [ G t ∣ S t = s ] = a ∑ q π ( s , a ) E [ G t ∣ S t = s , A t = a ] π ( a ∣ s ) Hence,
v π ( s ) = ∑ a π ( a ∣ s ) q π ( s , a ) (2) {\color{red}v_{\pi}(s)} = \sum_{a} \pi(a|s) {\color{red}q_{\pi}(s,a)} \tag{2} v π ( s ) = a ∑ π ( a ∣ s ) q π ( s , a ) ( 2 ) 再看贝尔曼公式
v π ( s ) = ∑ a π ( a ∣ s ) [ ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v π ( s ′ ) ] ⏟ q π ( s , a ) (3) v_{\pi}(s) = \sum_{a} \pi(a|s) \underbrace{\left[ \sum_{r} p(r|s, a) r + \gamma \sum_{s'} p(s'|s, a) v_{\pi}(s') \right]}_{{\color{red}q_{\pi}(s,a)}} \tag{3} v π ( s ) = a ∑ π ( a ∣ s ) q π ( s , a ) [ r ∑ p ( r ∣ s , a ) r + γ s ′ ∑ p ( s ′ ∣ s , a ) v π ( s ′ ) ] ( 3 ) By comparing (2) and (3), we have the action-value function as
q π ( s , a ) = ∑ r p ( r ∣ s , a ) r + γ ∑ s ′ p ( s ′ ∣ s , a ) v π ( s ′ ) (4) {\color{red}q_{\pi}(s,a)} = \sum_{r} p(r|s, a) r + \gamma \sum_{s'} p(s'|s, a) {\color{red}v_{\pi}(s')} \tag{4} q π ( s , a ) = r ∑ p ( r ∣ s , a ) r + γ s ′ ∑ p ( s ′ ∣ s , a ) v π ( s ′ ) ( 4 ) (2) and (4) are the two sides of the same coin :
(2) shows how to obtain state values from action values.
(4) shows how to obtain action values from state values.
