A safe multi-agent reinforcement learning algorithm using constraint update projection approach

Yang Liu

doi:10.1016/j.engappai.2025.111929

Return

A safe multi-agent reinforcement learning algorithm using constraint update projection approach

2025-09-01

PRE

DOI：10.1016/j.engappai.2025.111929

Original

Original request for help

Save

Abstract

En 中文

Traditional reinforcement learning has a major limitation, which is that they optimize agent’s policy purely for maximizing rewards. It completely ignore safety considerations. However, in certain critical engineering fields, ensuring safety is of utmost importance, otherwise it can cause incalculable losses. Therefore, this paper proposes a safe Multi-Agent Constrained Update Projection(MACUP) algorithm, which can safely control agents to complete tasks. We solve this problem from the perspective of policy constraint optimization. Firstly, we derive the new bounds of multi-agent policy performance difference based on a tighter general policy performance difference. It contains generalized advantage estimates, and we utilize these bounds as surrogate functions concerning the objective and constraints. Secondly, to address the coordination issue among multiple agents, we employ a multi-agent sequential policy update framework. Finally, we use a projection method to optimize policies, which has low computational complexity and does not require convex approximation of the surrogate function for solving. It can help us reduce errors. Finally, we have validated our algorithm in two different multi-agent safety environments, and the results show that it is able to satisfy safety constraints while achieving higher rewards.

Keywords:

safe reinforcement learning

multi-agent systems

policy constraint optimization

constrained update

projection method