Sample-Efficient Reinforcement Learning From Human Feedback via Information-Directed Sampling

Qi Han

doi:10.1109/TIT.2025.3598296

Return

Sample-Efficient Reinforcement Learning From Human Feedback via Information-Directed Sampling

2025-10-01

PRE

DOI：10.1109/TIT.2025.3598296

Original

Original request for help

Save

Abstract

En 中文

We study the problem of reinforcement learning from human feedback (RLHF), a critical problem in training large language models, from a theoretical perspective. Our main contribution is the design of novel sample-efficient RLHF algorithms based on information-directed sampling (IDS), an online decision-making principle inspired by information theory. Our algorithms maximize the sum of the value function and a mutual information term that encourages exploration of the unknown environment (which quantifies the information gained about the environment through observed human feedback data). To tackle the challenge of large state spaces and improve sample efficiency, we construct a simplified surrogate environment and introduce a novel distance measure (named the l(g)-distance), enabling our IDS-based algorithm to achieve a Bayesian regret upper bound of order O(H-3/2 root log (K(epsilon))T) , where H is the episode length, T is the number of episode and K(epsilon) is related to the covering number of the environment. Specializing to the tabular settings, this regret bound is of order(sic)(H-2 root SAT) , where S and A are the numbers of states and actions. Finally, we propose an Approximate-IDS algorithm that is computationally more efficient while maintaining nearly the same sample efficiency. The design principle of this approximate algorithm is not only effective in RLHF settings but also applicable to the standard RL framework. Moreover, our work showcases the value of information theory in reinforcement learning and in the training of large language models.

Keywords:

Bayes methods

Reinforcement learning

Approximation algorithms

Standards

Mutual information

Trajectory

Training

Random variables

Optimization

Large language models

reinforcement learning from human feedback