TVDO: Tchebycheff Value-Decomposition Optimization for Multiagent Reinforcement Learning

doi:10.1109/TNNLS.2024.3455422

TVDO: Tchebycheff Value-Decomposition Optimization for Multiagent Reinforcement Learning

2024-01-01

0

PRE

AI

DOI：10.1109/TNNLS.2024.3455422

Original

Original request for help

Save

Abstract

En 中文

In cooperative multiagent reinforcement learning (MARL), centralized training with decentralized execution (CTDE) has recently attracted more attention due to the physical demand. However, the most dilemma therein is the inconsistency between jointly-trained policies and individually executed actions. In this article, we propose a factorized Tchebycheff value-decomposition optimization (TVDO) method to overcome the trouble of inconsistency. In particular, a nonlinear Tchebycheff aggregation function is formulated to realize the global optimum by tightly constraining the upper bound of individual action-value bias, which is inspired by the Tchebycheff method of multiobjective optimization (MOO). We theoretically prove that, under no extra limitations, the factorized value decomposition with Tchebycheff aggregation satisfies the sufficiency and necessity of individual-global-max (IGM), which guarantees the consistency between the global and individual optimal action-value function. Empirically, in the climb and penalty game, we verify that TVDO precisely expresses the global-to-individual value decomposition with a guarantee of policy consistency. Meanwhile, we evaluate TVDO in the StarCraft multiagent challenge (SMAC) benchmark, and extensive experiments demonstrate that TVDO achieves a significant performance superiority over some SOTA MARL baselines.

Keywords:

Multiagent cooperative learning

reinforcement learning

Tchebycheff method

value decomposition

TVDO: Tchebycheff Value-Decomposition Optimization for Multiagent Reinforcement Learning

Abstract

Journal

Organization