Robust Multiobjective Reinforcement Learning Considering Environmental Uncertainties

doi:10.1109/TNNLS.2024.3397393

Robust Multiobjective Reinforcement Learning Considering Environmental Uncertainties

2024-01-01

0

PRE

AI

DOI：10.1109/TNNLS.2024.3397393

Original

Original request for help

Save

Abstract

En 中文

Numerous real-world decision or control problems involve multiple conflicting objectives whose relative importance (preference) is required to be weighed in different scenarios. While Pareto optimality is desired, environmental uncertainties (e.g., environmental changes or observational noises) may mislead the agent into performing suboptimal policies. In this article, we present a novel multiobjective optimization paradigm, robust multiobjective reinforcement learning (RMORL) considering environmental uncertainties, to train a single model that can approximate robust Pareto-optimal policies across the entire preference space. To enhance policy robustness against environmental changes, an environmental disturbance is modeled as an adversarial agent across the entire preference space via incorporating a zero-sum game into a multiobjective Markov decision process (MOMDP). Additionally, we devise an adversarial defense technique against observational perturbations, which ensures that policy variations, perturbed by adversarial attacks on state observations, remain within bounds under any specified preferences. The proposed technique is assessed in five multiobjective environments with continuous action spaces, showcasing its effectiveness through comparisons with competitive baselines, which encompass classical and state-of-the-art schemes.

Keywords:

Uncertainty

Optimization

Task analysis

Robustness

Perturbation methods

Noise

Reinforcement learning

Adversarial machine learning

decision and control

multiobjective optimization

reinforcement learning (RL)

Robust Multiobjective Reinforcement Learning Considering Environmental Uncertainties

Abstract

Journal

Organization