关键词: 强化学习, 自然策略梯度, 经验复用, 倒立摆控制

Abstract:

Though eNAC ( episodic Natural Actor-Critic) algorithm,an episode-based reinforcement learning control algorithm,is theoretically of excellent learning performance,it is inefficient in learning because many episodes are required to obtain a good policy. In order to solve this problem,a new algorithm named ER-eNAC,which introduces the episode reuse mechanism in eNAC algorithm,is proposed. In ER-eNAC,some of the past episodes are
reused in the estimation procedure of current natural policy gradient for the purpose of using the experience more efficiently,and the reused episodes are weighted in an exponential decay according to the number of policy updates that they have undergone for the purpose of describing their fitness to the current policy. The proposed algorithm is then applied to the inverted pendulum control. Simulated results show that,as compared with eNAC algorithm,ER-eNAC algorithm is more effective because it significantly reduces the number of episodes for learning and remarkably improves the learning efficiency.

Key words: reinforcement learning, natural policy gradient, experience reuse, inverted pendulum control