SIPA: a self-iterative preference alignment method for generative language models

Yongping Du

doi:10.1007/s10489-025-07079-9

Return

SIPA: a self-iterative preference alignment method for generative language models

2026-01-29

PRE

DOI：10.1007/s10489-025-07079-9

Original

Original request for help

Save

Abstract

En 中文

While the current preference optimization methods for aligning large language models have demonstrated promising performance, extensive reliance on large amounts of manually annotated preference data presents significant barriers for broader application. The acquisition of annotated data is costly and inherently subjective, bringing challenges to the model’s fairness, robustness and trustworthiness. We propose a Self-Iterative Preference Alignment (SIPA) method, integrating both On-policy and Off-policy optimization strategies to reduce this reliance. We investigate the crucial role of self-optimization in model alignment by conducting experiments on a biased dataset and the evaluations are designed using the LLM-as-Judge framework. The experimental results indicate that our method improves the preference alignment performance of the baseline model effectively. Specifically, it achieves better performance with only ten percent human-annotated preference data, demonstrating its strong potential for application in low-resource scenarios. The data distribution shifts during iterative training are analyzed by visualizations, highlighting SIPA’s effectiveness in enhancing model alignment with human preferences.

Keywords:

Large language models

Preference alignment

Self-iterative

Human annotation