arrow
返回

Active Preference Optimization for Sample Efficient RLHF

delete2026-01-01
delete0
PRE
AI
N
Nirjhar Das *
S
Souradip Chakraborty
A
Aldo Pacchiano
S
Sayak Ray Chowdhury
DOI:10.1007/978-3-032-06096-9_6delete
delete原文链接
delete分享
delete收藏
摘要

摘要

En 中文
Large Language Models (LLMs) aligned using Reinforcement Learning from Human Feedback (RLHF) have shown remarkable generation abilities in numerous tasks. However, collecting high-quality human preferences creates costly bottlenecks in practical deployments, and hence, training data are often budgeted. In these scenarios, it is crucial to collect training data (e.g., contexts, a pair of generations for each context, and a preference indicating which generation is better) carefully, yet most of the existing methods sample contexts uniformly at random from a given collection. Given this, under the Bradley-Terry-Luce preference model and with a small budget of training data, we show that uniform sampling of contexts could lead to a policy (i.e., an aligned model) that suffers a constant sub-optimality gap from the optimal policy. This highlights the need for an adaptive context sampling strategy for effective alignment under a small sample budget. To address this, we reformulate RLHF within the contextual preference bandit framework, treating generations as actions, and give a nearly complete characterization of the sub-optimality gap in terms of both lower and upper bounds. First, when the action set is a d-dimensional hypercube and the number of samples is T, we show an Omega(d/root T) lower bound. Next, we propose an algorithm, Active Preference Optimization (APO), that iteratively collects preferences for the most uncertain contexts. We show that the sub-optimality gap of the policy learned via APO matches the lower bound up to a log factor and a non-linearity constant. Finally, we perform experiments on practical datasets to validate APO's efficacy over existing methods, establishing it as a sample-efficient and cost-effective solution for LLM alignment.

期刊

M
MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES. RESEARCH TRACK, ECML PKDD 2025, PT V
IF:
0
论文数:
26
被引数:
3

机构

B
Boston University
学者数:
1.7K
论文数: 655
被引数: 2.4W
I
indian institute of science (iisc) - bangalore
学者数:
0
论文数: 237
被引数: 6.9K
U
university of maryland college park
学者数:
87
论文数: 82
被引数: 73
University System of Maryland 封面图
University System of Maryland
学者数:
6.5W
论文数: 5.6W
被引数: 8.2W
学者 查看更多机构
引用论文

引用论文

Bandit Algorithms强盗算法
err
IF0
err2020-07-04
err0
errOAAI
errTor Lattimore; Csaba Szepesvári
err分享
err收藏
err分享
err收藏
Active Preference-Based Learning of Reward Functions
err2017-07-12
err0
PREAI
errDorsa Sadigh; Anca Dragan; Shankar Sastry; Sanjit Seshia
err分享
err收藏