policy gradient详解（附代码）

1 引言

 policy gradient是强化学习中一种基于概率策略的方法。智能体通过与环境的交互获得特定时刻的状态信息，并直接给出下一步要采取各种动作的概率，然后根据该状态动作的策略分布采取下一步的行动，所以每种动作都有可能被选中，只是选中的概率性不同。智能体直接学习状态动作的策略分布，在强化学习的训练中，用神经网络来表示状态动作分布，给一个状态，就会输出该状态下的动作分布。强化学习算法直接对策略进行优化，使指定的策略能够获得最大的奖励。

2 policy gradient原理介绍

 考虑一个随机参数化的策略${\pi }_{\theta }$，强化学习主要目标是最大化期望回报$$J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[R(\tau )]$$其中$\tau =(s_0,a_0,s_1,a_1,\cdots ,s_{T+1})$，$s_i$和$a_i$分别表示第$i$时刻的状态和动作。$R(\tau )=\sum _{t=0}^T{r}_t$表示$T$时刻内的回报，$r_t$表示第$t$时刻的回报。通过梯度上升法优化策略即有$${\theta }_{k+1}={\theta }_k+\alpha {\nabla }_{\theta }J({\pi }_{\theta }){|}_{{\theta }_k}$$其中${\nabla }_{\theta }J({\pi }_{\theta })$表示策略梯度。策略梯度的具体导数形式的推导如下所示。给定策略${\pi }_{\theta }$，轨迹$\tau$的概率为$$P(\tau |\theta )={\rho }_0(s_0)\prod _{t=0}^{T}P(s_{t+1}|s_t,a_t){\pi }_{\theta }(a_t|s_t)$$其中${\rho }_0(\cdot )$表示状态分布。根据链式法则则有$${\nabla }_{\theta }P(\tau |\theta )=P(\tau |\theta ){\nabla }_{\theta }\log P(\tau |\theta )$$进一步可知轨迹的对数概率表示为$$\log P(\tau |\theta )=\log {\rho }_0(s_0)+\sum _{t=0}^T\left(\log P(s_{t+1}|s_t,a_t)+\log {\pi }_{\theta }(a_t|s_t)\right)$$因为${\rho }_0(s_0)$，$P(s_{t+1}|s_t,a_t)$与策略参数$\theta$无关 ，所以它们的梯度为$0$，进而可知轨迹的对数概率梯度表示为$$\begin{array}{rl}{\nabla }_{\theta }\log P(\tau |\theta )& ={\nabla }_{\theta }\log {\rho }_0(s_0)+\sum _{t=0}^T\left({\nabla }_{\theta }\log P(s_{t+1}|s_t,a_t)+{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\right)\\ & =\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\end{array}$$综上所述可得$$\begin{array}{rl}{\nabla }_{\theta }J({\pi }_{\theta })& ={\nabla }_{\theta }{\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[R(\tau )]\\ & ={\nabla }_{\theta }{\int }_{\tau }P(\tau |\theta )R(\tau )\\ & ={\int }_{\tau }{\nabla }_{\theta }P(\tau |\theta )R(\tau )\\ & ={\int }_{\tau }P(\tau |\theta ){\nabla }_{\theta }\log P(\tau |\theta )R(\tau )\\ & ={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}[{\nabla }_{\theta }\log P(\tau |\theta )R(\tau )]\\ & ={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)R(\tau )\right]\end{array}$$以上公式是期望的表示形式，可以通过对蒙特卡洛模拟采样均值来估计它的值。假定采样得到一个轨迹集合D={τi}i=1,⋯ ,N\mathcal{D}=\{\tau_i\}_{i=1,\cdots,N}D={τi​}i=1,⋯,N​，其中每个轨迹都是在策略${\pi }_{\theta }$下智能体与环境交互得到的，此时估计的策略梯度表示为$$\hat{g}=\frac{1}{|\mathcal{D}|}\sum _{\tau \in \mathcal{D}}\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)R(\tau )$$其中$|\mathcal{D}|$表示轨迹集合$\mathcal{D}$的元素个数。

3 EGLP引理

 根据策略梯度可以推导出一个中间结果为对数概率梯度的期望（Expected Grad-Log-Prob）引理。

EGLP引理： 假定$P_{\theta }$是随机变量$x$的参数化的概率分布，进而则有$${\mathbb{E}}_{x\sim P_{\theta }}[{\nabla }_{\theta }\log P_{\theta }(x)]=0$$

证明： 由概率分布的特性可知$${\int }_x{P}_{\theta }(x)=1$$对以上等式两边求梯度可知$${\nabla }_{\theta }{\int }_x{P}_{\theta }(x)={\nabla }_{\theta }1=0$$由对数导数可知$$\begin{array}{rl}0& ={\nabla }_{\theta }{\int }_x{P}_{\theta }(x)\\ & ={\int }_x\nabla P_{\theta }(x)\\ & ={\int }_x{P}_{\theta }(x){\nabla }_{\theta }\log P_{\theta }(x)\end{array}$$所以可知$${\mathbb{E}}_{x\sim P_{\theta }}[{\nabla }_{\theta }\log P_{\theta }(x)]=0$$

4 policy gradient改进版

 已知策略梯度的表达式为$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)R(\tau )\right]$$由以上公式可知，在一个状态动作轨迹里，当执行下一步动作的时，得到的回报是固定的，即为总回报$R_{\tau }$，这有背与常识。根据经验可知，一个智能体应该根据其执行动作之后带来结果再对当前的行为进行更新，跟行动之前的结果好坏无关，所以可以得到改进版的policy gradient的公式为$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\sum _{t^{\prime }=t}^{T}R(s_{t^{\prime }},a_{t^{\prime }},s_{t^{\prime }+1})\right]$$以上公式表明行动只会根据采取行动后获得的奖励来加强，其中$${\hat{R}}_t=\sum _{t^{\prime }=t}^{T}R(s_{t^{\prime }},a_{t^{\prime }},s_{t^{\prime }+1})$$该公式表示在$t$时刻之后的奖励回报。

 根据EGLP引理可推知，对于任意只依赖状态的函数$b(\cdot )$，可得$${\mathbb{E}}_{a_t\sim {\pi }_{\theta }}[{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)b(s_t)]=0$$这可以在改进版的策略梯度公式中随意加上或者减去函数$b(\cdot )$，即有$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\left(\sum _{t^{\prime }=t}^{T}R(s_{t^{\prime }},a_{t^{\prime }},s_{t^{\prime }+1})-b(s_t)\right)\right]$$其中$b(s)$被称作baseline函数。一般情况下baseline函数会选择状态值函数$V^{\pi }(s)$。

5 policy gradient其它形式

 policy graident的通用形式如下公式所示$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t){\Phi }_t\right]$$根据${\Phi }_t$函数的不同可以将policy gradient种类划分为如下形式

• 轨迹$\tau$总回报函数：$${\Phi }_t=R(\tau )$$
• 轨迹$\tau$在$t$时刻后回报函数：$${\Phi }_t=\sum _{t^{\prime }=t}^{T}R(s_{t^{\prime }},a_{t^{\prime }},s_{t^{\prime }+1})$$
• 状态值函数：$${\Phi }_t=V^{\pi }(s)={\mathbb{E}}_{\tau \sim \pi }[R(\tau )|s_0=s]$$
• 动作状态值函数：$${\Phi }_t=Q^{\pi }(s,a)={\mathbb{E}}_{\tau \sim \pi }[R(\tau )|s_0=s,a_0=a]$$
• 优势函数：$${\Phi }_t=A^{\pi }(s_t,a_t)=Q^{\pi }(s_t,a_t)-V^{\pi }(s_t)$$

6 程序代码

 policy gradient的pytorch代码实现如下所示，此代码实现了的policy gradient是以下的形式$${\nabla }_{\theta }J({\pi }_{\theta })={\mathbb{E}}_{\tau \sim {\pi }_{\theta }}\left[\sum _{t=0}^T{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)\sum _{t^{\prime }=t}^{T}R(s_{t^{\prime }},a_{t^{\prime }},s_{t^{\prime }+1})\right]$$在以下文件RL_template.py中进行了实现。

import numpy as np  
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical


class Policy(nn.Module):
    def __init__(self, in_features = 4, hid_features = 128, out_features = 2, pro = 0.6):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(in_features,hid_features)
        self.dropout = nn.Dropout(p = pro)
        self.fc2 = nn.Linear(hid_features, out_features)
 
    def forward(self, x):
        x = self.fc1(x)
        x = self.dropout(x)
        x = F.relu(x)
        action_scores = self.fc2(x)
        return F.softmax(action_scores,dim=1)


class PolicyGradient(object):
	def __init__(
		self, 
		policy_net,
		learning_rate = 0.01, 
		reward_decay = 0.95
	):
		
		self.policy_net = policy_net
		self.lr = learning_rate
		self.gamma = reward_decay

		self.ep_ss = []
		self. ep_as= []
		self.ep_rs = []
		self.ep_log_pros = []

	def choose_action(self, state):
		state = torch.from_numpy(state).float().unsqueeze(0)
		probs = self.policy_net(state)
		m = Categorical(probs)
		action = m.sample()
		# m.log_prob(action) <===> probs.log()[0][action.item()].unsqueeze(0)
		self.ep_log_pros.append(m.log_prob(action)) 
		return action.item()


	def store_transition(self, s, a, r):
		self.ep_ss.append(s)
		self.ep_as.append(a)
		self.ep_rs.append(r)


	def eposide_learning(self):
		optimizer = optim.Adam(self.policy_net.parameters(),lr=1e-2)
		eps = np.finfo(np.float32).eps.item()  
		R = 0
		policy_loss = []
		returns = []

		for r in self.ep_rs[::-1]:
			R = r + self.gamma * R
			returns.insert(0, R)  

		returns = torch.tensor(returns)
		returns = (returns - returns.mean()) / (returns.std() + eps)

		for log_prob, R in zip(self.ep_log_pros, returns):
			policy_loss.append(-log_prob * R)

		policy_loss = torch.cat(policy_loss).sum()

		optimizer.zero_grad()
		policy_loss.backward()
		optimizer.step()

		del self.ep_rs[:]
		del self.ep_log_pros[:]
		del self.ep_as[:]
		del self.ep_ss[:]

 以下代码可以针对不用游戏环境进行强化学习。

import argparse
from RL_template import PolicyGradient, Policy
import gym


parser = argparse.ArgumentParser(description='Pytorch REINFORCE example')
parser.add_argument('--render',action='store_false')
parser.add_argument('--episodes', type=int, default=1000)
parser.add_argument('--steps_per_episode', type=int, default=100)
parser.add_argument('--gamma', type=float, default=0.99)
parser.add_argument('--seed',type=int, default=543)
args = parser.parse_args()

# env = gym.make('CartPole-v1')
env = gym.make('MountainCar-v0')


policy_net = Policy(
		in_features = env.observation_space.shape[0],
		out_features = env.action_space.n
		)

print(env.action_space)
print(env.observation_space)
print(env.observation_space.high)
print(env.observation_space.low)


RL = PolicyGradient(
	learning_rate = 0.002,
	policy_net = policy_net)

for episode in range(args.episodes):

	state, ep_reward = env.reset(), 0

	while True:
	# for t in range(args.steps_per_episode):
		if args.render:
			env.render()

		action = RL.choose_action(state)

		state, reward, done, info = env.step(action)

		RL.store_transition(state, action, reward)

		ep_reward += reward

		if done == True:
			break

	RL.eposide_learning()

	print('Episode {}\tLast reward: {:.2f}'.format(episode, ep_reward))