.png){kind=icon}
返回
Policy Evaluation in Continuous MDPs With Efficient Kernelized Gradient Temporal Difference
.png){kind=icon}
OA
AI
DOI：10.1109/TAC.2020.3029315.png)
.png){kind=icon}
.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
We consider policy evaluation in infinite-horizon discounted Markov decision problems with continuous compact state and action spaces. We reformulate this task as a compositional stochastic program with a function-valued decision variable that belongs to a reproducing kernel Hilbert space (RKHS). We approach this problem via a new functional generalization of stochastic quasi-gradient methods operating in tandem with stochastic sparse subspace projections. The result is an extension of gradient temporal difference learning that yields nonlinearly parameterized value function estimates of the solution to the Bellman evaluation equation. We call this method parsimonious kernel gradient temporal difference learning. Our main contribution is a memory-efficient nonparametric stochastic method guaranteed to converge exactly to the Bellman fixed point with probability 1 with attenuating step-sizes under the hypothesis that it belongs to the RKHS. Further, with constant step-sizes and compression budget, we establish mean convergence to a neighborhood and that the value function estimates have finite complexity. In the Mountain Car domain, we observe faster convergence to lower Bellman error solutions than existing approaches with a fraction of the required memory.
Keyword:
Kernel
Complexity theory
Markov processes
Hilbert space
Convergence
Memory management
Automobiles
Iterative learning control
markov processes
optimization methods
stochastic systems
AI总结
对已上传原文的论文进行重点信息的提取，主要内容包括：简要概述、研究摘要、背景介绍、关键亮点、图文解析、展望与总结。
期刊
IF:
7
论文数:
1.3W
被引数:
6.7W
机构
引用论文
.png){kind=icon}
查看更多内容