Natural Gradient Works Efficiently in Learning

doi:10.1162/089976698300017746

Natural Gradient Works Efficiently in Learning

1998-02-01

0

PRE

AI

DOI：10.1162/089976698300017746

原文链接

原文求助

摘要

En 中文

When a parameter space has a certain underlying structure, the ordinary gradient of a function does not represent its steepest direction, but the natural gradient does. Information geometry is used for calculating the natural gradients in the parameter space of perceptrons, the space of matrices (for blind source separation), and the space of linear dynamical systems (for blind source deconvolution). The dynamical behavior of natural gradient online learning is analyzed and is proved to be Fisher efficient, implying that it has asymptotically the same performance as the optimal batch estimation of parameters. This suggests that the plateau phenomenon, which appears in the backpropagation learning algorithm of multilayer perceptrons, might disappear or might not be so serious when the natural gradient is used. An adaptive method of updating the learning rate is proposed and analyzed.

Natural Gradient Works Efficiently in Learning

摘要

AI总结

期刊

机构

引用论文