通过精选偏好数据提升大模型对齐效果，10%数据胜过全量训练。
Less is More: Improving LLM Alignment via Preference Data Selection
- 基于边际最大化原理筛选高价值偏好数据，减少噪声干扰。
- 仅用Ultrafeedback的10%数据，在AlpacaEval2上提升3%-8%。
- 适用于多模型与迭代训练，显著降低高质量数据依赖。
直接偏好优化（DPO）已成为对齐大语言模型与人类偏好的有效方法。现有工作主要关注目标函数的改进，而本文从常被忽视但至关重要的数据选择角度入手。针对噪声数据导致的参数收缩问题，提出一种新的边际最大化原则用于DPO训练的数据集筛选。为缓解不同奖励模型带来的噪声，引入贝叶斯聚合方法，统一整合外部与隐式边际信息，生成单一偏好概率。在多种设置下的实验证明该方法具有极高的数据效率：仅使用Ultrafeedback数据集的10%，在Llama、Mistral和Qwen系列模型上于AlpacaEval2基准上实现3%至8%的性能提升。此外，该方法可无缝扩展至迭代DPO，仅需25%在线数据即获得约3%的增益，揭示了传统高质量数据构建方式中存在显著冗余。这些结果凸显了数据选择策略在推进偏好优化中的巨大潜力。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences. While prior work mainly extends DPO from the aspect of the objective function, we instead improve DPO from the largely overlooked but critical aspect of data selection. Specifically, we address the issue of parameter shrinkage caused by noisy data by proposing a novel margin-maximization principle for dataset curation in DPO training. To further mitigate the noise in different reward models, we propose a Bayesian Aggregation approach that unifies multiple margin sources (external and implicit) into a single preference probability. Extensive experiments in diverse settings demonstrate the consistently high data efficiency of our approach. Remarkably, by using just 10\% of the Ultrafeedback dataset, our approach achieves 3\% to 8\% improvements across various Llama, Mistral, and Qwen models on the AlpacaEval2 benchmark. Furthermore, our approach seamlessly extends to iterative DPO, yielding a roughly 3\% improvement with 25\% online data, revealing the high redundancy in this presumed high-quality data construction manner. These results highlight the potential of data selection strategies for advancing preference optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。