Beyond backpropagate through time: Efficient model-based training through time-splitting

doi:10.1002/int.22928

Beyond backpropagate through time: Efficient model-based training through time-splitting

2022-05-24

0

PRE

AI

DOI：10.1002/int.22928

原文链接

原文求助

摘要

En 中文

Model-based policy gradient (MBPG) has been employed to seek an approximate solution to the optimal control problem. However, there is coupling between adjacent states due to temporal dependencies, making the training time grow linearly with the time horizon. This paper reshapes the training process of MBPG with the time-splitting technique to establish a time-independent algorithm called Training Through Time-Splitting (T3S). First, copy the coupled variables to obtain two independent variables. Meanwhile, an extra variable together with an equivalence constraint is introduced for problem consistency. Then, the transformed problem divides into subproblems with carefully derived loss functions. Subproblems own decoupled variables and shared policy networks, which means they can be optimized concurrently. Guided by the algorithm design, this paper further proposes an asynchronous parallel training scheme to accelerate training efficiency. Numerical simulation shows that the T3S algorithm outperforms the MBPG algorithm by 83.6% in wall-clock time with a trajectory tracking task.

Keyword:

model-based policy gradient

optimal control

parallel training

reinforcement learning

time-splitting

Beyond backpropagate through time: Efficient model-based training through time-splitting

摘要

期刊

机构