Continual Reinforcement Learning Formulation for Zero-Sum Game-Based Constrained Optimal Tracking

Behzad Farzanegan

doi:10.1109/TSMC.2023.3299556

Continual Reinforcement Learning Formulation for Zero-Sum Game-Based Constrained Optimal Tracking

2023-12-01

5

PRE

AI

DOI：10.1109/TSMC.2023.3299556

Original

Original request for help

Save

Abstract

En 中文

This study provides a novel reinforcement learning-based optimal tracking control of partially uncertain nonlinear discrete-time (DT) systems with state constraints using zero-sum game (ZSG) formulation. To address optimal tracking, a novel augmented system consisting of tracking error and its integral value, along with an uncertain desired trajectory, is constructed. A barrier function (BF) with a tradeoff factor is incorporated into the cost function to keep the state trajectories to remain within a compact set and to balance safety with optimality. Next, by using the modified value functional, the ZSG formulation is introduced wherein an actor-critic neural network (NN) framework is employed to approximate the value functional, optimal control input, and worst disturbance. The critic NN weights are tuned once at the sample instants and then iteratively within sampling instants. Using control input errors, the actor NN weights are adjusted once a sampling instant. The concurrent learning term in the critic weight tuning law overcomes the need for the persistency excitation (PE) condition. Further, a weight consolidation scheme is incorporated into the critic update law to attain lifelong learning by overcoming catastrophic forgetting. Finally, a numerical example supports the analytical claims.

Keywords:

Barrier Lyapunov function

experience replay

hybrid learning

lifelong learning

optimal tracking control

safety

zero-sum game (ZSG)

Continual Reinforcement Learning Formulation for Zero-Sum Game-Based Constrained Optimal Tracking

Abstract

Journal

Organization