Evolving Consensus Control With Iterative Q-Learning for Discrete-Time Multi-Agent Systems

doi:10.1109/tase.2026.3737165

Evolving Consensus Control With Iterative Q-Learning for Discrete-Time Multi-Agent Systems

2026-09-24

0

PRE

AI

DOI：10.1109/tase.2026.3737165

Original

Original request for help

Save

Abstract

En 中文

This article investigates the value-iteration-based Q-learning (VIQL) scheme for discrete-time nonlinear multi-agent systems with unknown dynamics and constrained control inputs, aiming to address the optimal consensus control problem in multi-agent systems arising from practical engineering applications. First, an iterative Q-learning framework is formulated to apply in the multi-agent systems characterized by unknown dynamics. Next, rigorous investigations are conducted to examine the convergence property exhibited by the sequence of iterative Q-functions. Then, systematic investigations of policy admissibility and system stability are conducted. An admissible criterion is established, providing a theoretical foundation for online evolving control of the VIQL scheme in multi-agent systems under input constraints. Furthermore, based on the monotonicity of the iterative Q-function sequence, an online evolving control algorithm is devised for the optimal consensus control problem. Finally, by including several representative examples of multi-agent systems with physical backgrounds, the excellent performance of online evolving control is convincingly illustrated with the developed algorithm. Note to Practitioners—This article is motivated by practical distributed control problems in real engineering systems, including vehicle platooning, coordinated robotic manipulators, and other networked automation platforms, where accurate system models are often unavailable and the agents must achieve coordinated behavior through local interactions. In these applications, the control inputs are typically subject to asymmetric physical bounds, which makes direct implementation of conventional optimal control methods difficult under realistic operating conditions. To address this issue, the proposed online VIQL framework provides a model-free learning mechanism that can acquire coordinated control policies directly from data while ensuring that the actual control commands remain feasible under the given actuator constraints. The main advantage of the developed method is that it supports online policy evolution without requiring prior system identification or offline redesign, which is particularly suitable for nonlinear multi-agent systems with unknown dynamics, coupling effects, and time-varying operating environments. The simulation studies on vehicle longitudinal platooning and robotic manipulator systems further illustrate that the proposed framework can achieve coordinated motion safely and effectively in representative engineering scenarios, where both performance improvement and input admissibility must be guaranteed simultaneously.

Keywords:

Adaptive dynamic programming

multi-agent systems

online evolving control

optimal consensus control

Q-learning

Evolving Consensus Control With Iterative Q-Learning for Discrete-Time Multi-Agent Systems

Abstract

Journal

Organization

Cited Papers