Proximal policy optimization with reward-based prioritization

Mingsheng Zheng

doi:10.1016/j.eswa.2025.127659

2025-07-01

PRE

DOI：10.1016/j.eswa.2025.127659

原文链接

原文求助

摘要

En 中文

PPO（Proximal Policy Optimization）算法是一种基于策略优化的深度强化学习算法，已取得卓越成果并得到广泛应用。尽管PPO算法广受欢迎，但其存在若干显著缺陷，包括对超参数的敏感性、收敛速度慢以及探索能力有限。近期研究中，代码级优化被提出以提升训练效率并取得了良好性能。本文提出了一种基于奖励优先级的近端策略优化算法（RP-PPO），该算法根据奖励对不同经验赋予不同优先级以更新策略，并寻找获得最高平均奖励的模型。我们还应用了包括归一化奖励和双学习率衰减等若干辅助技术来优化算法。最后，我们在Gym环境中的四个问题领域进行了一系列实验和测试，以证明我们算法的优越性。

Keyword:

Reinforcement learning

Proximal policy optimization

Prioritization

Policy gradient

Actor-critic

期刊

Expert Systems with Applications

IF:

7.5

论文数:

3.0W

被引数:

10.2W

机构

暂无机构信息

引用论文

Prioritized Experience Replay based on Multi-armed Bandit

EXPERT SYSTEMS WITH APPLICATIONS

IF7.5

2022-03-01

PREAI

Liu, Ximing; Zhu, Tianqing; Jiang, Cuiqing; Ye, Dayong; Zhao, Fuqing

Sampling diversity driven exploration with state difference guidance

EXPERT SYSTEMS WITH APPLICATIONS

IF7.5

2022-10-01

PREAI

Lu, Jiayi; Han, Shuai; Lu, Shuai; Kang, Meng; Zhang, Junwei

Deep Reinforcement Learning with Double Q-Learning 基于双Q学习的深度强化学习

Proceedings of the AAAI Conference on Artificial Intelligence

IF0

2016-03-02

OAAI

Hado Van Hasselt; Arthur Guez; David Silver

Deep multi-agent reinforcement learning for multi-level preventive maintenance in manufacturing systems 面向制造系统多级预防性维护的深度多智能体强化学习

EXPERT SYSTEMS WITH APPLICATIONS

IF7.5

2022-04-01

PREAI

Su, Jianyu; Huang, Jing; Adams, Stephen; Chang, Qing; Beling, Peter A.

Mastering Complex Control in MOBA Games with Deep Reinforcement Learning

Proceedings of the AAAI Conference on Artificial Intelligence

IF0

2020-04-03

OAAI

Deheng Ye; Zhao Liu; Mingfei Sun; Bei Shi; Peilin Zhao; Hao Wu; Hongsheng Yu; Shaojie Yang; Xipeng Wu; Qingwei Guo; Qiaobo Chen; Yinyuting Yin; Hao Zhang; Tengfei Shi; Liang Wang; Qiang Fu; Wei Yang; Lanxiao Huang

Application of deep reinforcement learning to intrusion detection for supervised problems 深度强化学习在监督问题入侵检测中的应用

EXPERT SYSTEMS WITH APPLICATIONS

IF7.5

2020-03-01

177

OAAI

Lopez-Martin, Manuel; Carro, Belen; Sanchez-Esguevillas, Antonio

Human-level control through deep reinforcement learning 通过深度强化学习实现人类层面的控制

Nature

IF0

2015-02-25

PREAI

Volodymyr Mnih; Koray Kavukcuoglu; David Silver; Andrei A. Rusu; Joel Veness; Marc G. Bellemare; Alex Graves; Martin Riedmiller; Andreas K. Fidjeland; Georg Ostrovski; Stig Petersen; Charles Beattie; Amir Sadik; Ioannis Antonoglou; Helen King; Dharshan Kumaran; Daan Wierstra; Shane Legg; Demis Hassabis

Actor-critic with familiarity-based trajectory experience replay

Information Sciences

IF0

2022-01-01

PREAI

Xiaoyu Gong; Jiayu Yu; Shuai Lü; Hengwei Lu

Reinforcement learning algorithms: A brief survey 强化学习算法: 简要综述

EXPERT SYSTEMS WITH APPLICATIONS

IF7.5

2023-11-01

PREAI

Shakya, Ashish Kumar; Pillai, Gopinatha; Chakrabarty, Sohom