.png){kind=icon}
返回
SIPA: a self-iterative preference alignment method for generative language models
.png){kind=icon}
PRE
AI
DOI：10.1007/s10489-025-07079-9.png)
.png){kind=icon}
原文求助.png){kind=icon}
分享.png){kind=icon}
收藏
摘要
En 中文
While the current preference optimization methods for aligning large language models have demonstrated promising performance, extensive reliance on large amounts of manually annotated preference data presents significant barriers for broader application. The acquisition of annotated data is costly and inherently subjective, bringing challenges to the model’s fairness, robustness and trustworthiness. We propose a Self-Iterative Preference Alignment (SIPA) method, integrating both On-policy and Off-policy optimization strategies to reduce this reliance. We investigate the crucial role of self-optimization in model alignment by conducting experiments on a biased dataset and the evaluations are designed using the LLM-as-Judge framework. The experimental results indicate that our method improves the preference alignment performance of the baseline model effectively. Specifically, it achieves better performance with only ten percent human-annotated preference data, demonstrating its strong potential for application in low-resource scenarios. The data distribution shifts during iterative training are analyzed by visualizations, highlighting SIPA’s effectiveness in enhancing model alignment with human preferences.
Keyword:
Large language models
Preference alignment
Self-iterative
Human annotation
期刊
IF:
3.5
论文数:
7.6K
被引数:
1.7W
机构
引用论文
Zephyr: direct distillation of LM alignment [large language model]Zephyr：大型语言模型[large language model]对齐的直接蒸馏.png)
OAAI
IF0
OAAI分享收藏Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluation事实性对齐：通过自我评估减轻LLMs中的幻觉
PREAI
分享收藏
查看更多内容