.png){kind=icon}
.png){kind=icon}
Return
Gradient compensation traces based temporal difference learning
.png){kind=icon}
PRE
AI
DOI：10.1016/j.neucom.2021.02.042.png)
.png){kind=icon}
Original request for help.png){kind=icon}
Share.png){kind=icon}
Save
Abstract
En 中文
For online updates and data efficiency, forward-view algorithms are transformed into backward-views, such as temporal difference learning (TD) and its control versions, by eligibility traces. Existing researches on eligibility traces, such as TD(A) and true-online TD(A), mainly focus on the equivalence between forward-views and backward-views. However, the choice of A refers to the time scope of the credit assignment, and a small A accelerates the decay of credit over the time. This paper takes a different implementation of the backward-view named gradient compensation traces (GCT). GCT compensates the difference between a bootstrapping estimated gradient and the true gradient online to remove the extra decay of the credit. Based on GCT, the corresponding temporal difference learning (gradient compensation TD, GCTD) is proved to converge conditionally. The sensitivity of GCTD's hyper-parameter is analyzed in the nonlinear long-corridor and linear random-walk task. The proposed algorithm is comparable with true-online TD(A) in the basic Mountain Car task, and outperforms the baselines in the reward sparse setting. (c) 2021 Elsevier B.V. All rights reserved.
Keywords:
Reinforcement learning
Eligibility traces
Value iteration
Temporal difference learning
AI Summary
Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.