Authentic Boundary Proximal Policy Optimization

doi:10.1109/TCYB.2021.3051456

Authentic Boundary Proximal Policy Optimization

2022-09-01

25

PRE

AI

DOI：10.1109/TCYB.2021.3051456

Original

Original request for help

Save

Abstract

En 中文

In recent years, the proximal policy optimization (PPO) algorithm has received considerable attention because of its excellent performance in many challenging tasks. However, there is still a large space for theoretical explanation of the mechanism of PPO's horizontal clipping operation, which is a key means to improve the performance of PPO. In addition, while PPO is inspired by the learning theory of trust region policy optimization (TRPO), the theoretical connection between PPO's clipping operation and TRPO's trust region constraint has not been well studied. In this article, we first analyze the effect of PPO's clipping operation on the objective function of conservative policy iteration, and strictly give the theoretical relationship between PPO and TRPO. Then, a novel first-order policy gradient algorithm called authentic boundary PPO (ABPPO) is proposed, which is based on the authentic boundary setting rule. To ensure the difference between the new and old policies is better kept within the clipping range, by borrowing the idea of ABPPO, we proposed two novel improved PPO algorithms called rollback mechanism-based ABPPO (RMABPPO) and penalized point policy difference-based ABPPO (P3DABPPO), which are based on the ideas of rollback clipping and penalized point policy difference, respectively. Experiments on the continuous robotic control tasks implemented in MuJoCo show that our proposed improved PPO algorithms can effectively improve the learning stability and accelerate the learning speed compared with the original PPO.

Keywords:

Task analysis

Linear programming

Optimization

Robots

Games

Reinforcement learning

Neural networks

Authentic boundary

penalized point policy difference

proximal policy optimization (PPO)

reinforcement learning (RL)

rollback clipping

Authentic Boundary Proximal Policy Optimization

Abstract

AI Summary

Journal

Organization