Combined data augmentation framework for generalizing deep reinforcement learning from pixels

doi:10.1016/j.eswa.2024.125810

Return

Combined data augmentation framework for generalizing deep reinforcement learning from pixels

2025-03-01

PRE

DOI：10.1016/j.eswa.2024.125810

Original

Original request for help

Save

Abstract

En 中文

The inherent generalization problem in reinforcement learning (RL) is exacerbated when dealing with highdimensional visual input. Data augmentation is a promising method to improve sample efficiency and generalization capability in RL. However, due to the sensitivity of RL training, the plain use of data augmentation can introduce more variability into the training process, making the optimization process more complex, leading to reduced sample efficiency, unstable training, and further deterioration of generalization performance. We propose a Combined Data Augmentation (CDA) framework that applies all data augmentation operators to visual RL. The CDA framework divides data augmentation methods into three data streams: no augmentation, pixel-level augmentation, and spatial-level augmentation. By treating each data stream separately, the CDA framework provides the visual RL pipeline with slight hardness and rich diversity data augmentation methods. It introduces different inductive biases for the agent, which can improve the sample efficiency while satisfying the smoothness assumption, i.e., ensuring that small changes in input data lead to small changes in model predictions, and enhancing the generalization performance, thus extending the highperformance policy to new environments that have not been seen before. The CDA framework is extensively evaluated on DeepMind Control Suite and corresponding generalization tasks. Experimental results show that CDA achieves better or comparable asymptotic performance in all training environments, and exhibits higher sample efficiency. Additionally, CDA outperforms other baseline methods in terms of generalization performance on 6 out of 10 tasks.

Keywords:

Reinforcement learning

Data augmentation

Generalization

Sample efficiency