Generative Adversarial Soft Actor-Critic

Hyo-Seok Hwang

doi:10.1109/TNNLS.2024.3493113

Generative Adversarial Soft Actor-Critic

2024-01-01

0

PRE

AI

DOI：10.1109/TNNLS.2024.3493113

Original

Original request for help

Save

Abstract

En 中文

In deep reinforcement learning (RL), learning stochastic and multimodal policies are crucial for various tasks, but most continuous control algorithms model the policy using a deterministic or unimodal Gaussian distribution. Despite being designed to inherently learn a stochastic policy under the maximum entropy RL framework, soft actor-critic (SAC) also uses a factorized Gaussian policy for tractable optimization, which not only restricts the expressiveness of the policy but also ignores correlations among the components of the action vector. In this article, we revisit the approach of employing normalizing flow for SAC policy, justified by the change of variable theorem, and then propose a state-dependent nonvolume preserving (SD-NVP) architecture suitable for SAC learning. In addition, we introduce a generative adversarial SAC (GASAC) that implicitly defines and optimizes various divergences without calculating the normalization constant through a generative adversarial loss. Experimental results on multigoal environment and MuJoCo continuous control tasks suite demonstrate that GASAC model multimodal policy and learn policy more stably in terms of cumulative return.

Keywords:

Entropy

Mathematical models

Generative adversarial networks

Training

Stochastic processes

Vectors

Optimization

Measurement

Computational modeling

Predictive models

Continuous control

deep learning

deep reinforcement learning (RL)

generative model

Generative Adversarial Soft Actor-Critic

Abstract

Journal

Organization

Cited Papers