提出四种智能加权策略，提升多参考模型微调效果。
Intelligently Weighting Multiple Reference Models for Direct Preference Optimization of LLMs
- 设计四种加权方法，基于验证数据或在线估计优化参考模型权重
- 在多个数据集上，新方法偏好准确率全面超越现有方案
- 意外发现单参考DPO反而优于多参考，挑战了多参考方法的实用性
微调是使大语言模型与人类偏好对齐的关键。多参考偏好优化（MRPO）在直接偏好优化（DPO）基础上，通过偏好数据微调模型，并以多个参考模型的混合作为正则项，以利用其综合优势。然而，现有参考权重设定方式缺乏理论依据，导致性能不可靠。为此，我们提出四种新权重策略：两种离线方法利用保留验证信号；一种在线方法采用滑动窗口估计器减少过拟合；另一种将参考加权建模为$K$-臂老虎机问题，使用汤普森采样。实验使用Qwen2.5-0.5B作为策略模型，七个来自Llama、Mistral、Qwen、Yi和Phi系列（0.5B-14B）的参考模型，在UltraFeedback和SafeRLHF数据集上验证。结果表明，所有四种策略均优于当前MRPO权重方法。更令人惊讶的是，仅用7个参考模型中的任意6个进行单参考DPO，其表现始终优于所有多参考方法，质疑了多参考方法的实际价值。
原文摘要 · Abstract (English)
Fine-tuning is integral for aligning large language models (LLMs) with human preferences. Multiple-Reference Preference Optimization (MRPO) builds on Direct Preference Optimization (DPO) by fine-tuning LLMs on preference datasets while regularizing the policy towards a mixture of reference models to leverage their collective desirable properties. However, current methods for setting the reference weights are ad-hoc and statistically unsound, leading to unreliable performance. To address this, we introduce four new weighting strategies: two offline methods that leverage held-out validation signal; one online method that uses a sliding-window estimator to reduce overfitting; and an online method that treats reference weighting as a $K$-armed bandit via Thompson Sampling. Experiments using Qwen2.5-0.5B as the policy model and seven reference models from the Llama, Mistral, Qwen, Yi, and Phi families (0.5B-14B each) show that all 4 of our strategies outperform the current MRPO weighting methods on UltraFeedback and SafeRLHF in preference accuracy. More thought-provokingly, however, we find that single-reference DPO, using any of 6 out of 7 references, consistently outperforms all tested multiple-reference approaches -- calling into question the practical appeal of multiple-reference approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。