Module-based reinforcement learning: Experiments with a real robot

Zsolt Kalmár

doi:10.1023/A:1007440607681

Module-based reinforcement learning: Experiments with a real robot

1998-01-01

23

OA

AI

DOI：10.1023/A:1007440607681

Original

Save

View PDF

Abstract

En 中文

The behavior of reinforcement learning (RL) algorithms is best understood in completely observable, discrete-rime controlled Markov chains with finite state and action spaces. In contrast, robot-learning domains are inherently continuous both in time and space, and moreover are partially observable. Here we suggest a systematic approach to solve such problems in which the available qualitative and quantitative knowledge is used to reduce the complexity of learning task. The steps of the design process are to: i) decompose the task into subtasks using the qualitative knowledge at hand; ii) design local controllers to solve the subtasks using the available quantitative knowledge and iii) learn a coordination of these controllers by means of reinforcement learning. It is argued that the approach enables fast, semi-automatic, but still high quality robot-control as no fine-tuning of the local controllers is needed. The approach was verified on a non-trivial real-life robot task. Several RL algorithms were compared by ANOVA and it was found that the model-based approach worked significantly better than the model-free approach. The learnt switching strategy performed comparably to a handcrafted version. Moreover, the learnt strategy seemed to exploit certain properties of the environment which were not foreseen in advance, thus supporting the view that adaptive algorithms are advantageous to non-adaptive ones in complex environments.

Keywords:

reinforcement learning

module-based RL

robot learning

problem decomposition

Markovian Decision Problems

feature space

subgoals

local control

switching control

Module-based reinforcement learning: Experiments with a real robot

Abstract

AI Summary

Journal

Organization