Importance sampling policy gradient algorithms in reproducing kernel Hilbert space

doi:10.1007/s10462-017-9579-x

Return

Importance sampling policy gradient algorithms in reproducing kernel Hilbert space

2017-10-10

PRE

DOI：10.1007/s10462-017-9579-x

Original

Original request for help

Save

Abstract

En 中文

Modeling policies in reproducing kernel Hilbert space (RKHS) offers a very flexible and powerful new family of policy gradient algorithms called RKHS policy gradient algorithms. They are designed to optimize over a space of very high or infinite dimensional policies. As a matter of fact, they are known to suffer from a large variance problem. This critical issue comes from the fact that updating the current policy is based on a functional gradient that does not exploit all old episodes sampled by previous policies. In this paper, we introduce a generalized RKHS policy gradient algorithm that integrates the following important ideas: (i) policy modeling in RKHS; (ii) normalized importance sampling, which helps reduce the estimation variance by reusing previously sampled episodes in a principled way; and (iii) regularization terms, which avoid updating the policy too over-fit to sampled data. In the experiment section, we provide an analysis of the proposed algorithms through bench-marking domains. The experiment results show that the proposed algorithm can still enjoy a powerful policy modeling in RKHS and achieve more data-efficiency.

Keywords:

Reproducing kernel Hilbert space

Policy search

Reinforcement learning

Importance sampling

Policy gradient

Non-parametric

AI Summary

Key information extracted from the uploaded paper, including a brief overview, abstract, background, key highlights, visual analysis, and future outlook.