只对关键词优化，提升大模型对齐效率与效果
Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
- 挑选对齐贡献大的关键词进行优化
- 在Arena-Hard上比标准方法提升12.3%得分
- 高质量参考模型能显著提升选择准确性
大语言模型后训练对齐面临挑战，因并非所有词元都同等影响性能。本文提出一种选择性对齐策略，通过当前策略与参考模型间的词元级对数概率差异，识别高影响力词元并优先优化。该方法降低计算开销，提升对齐精度。实验表明，更强的参考模型能显著提高词元选择准确率和整体优化效果。在Arena-Hard和MT-Bench等基准上，所提的Selective-DPO方法优于标准DPO与基于蒸馏的基线。结果强调了词元级优化与参考模型选择的重要性。
原文摘要 · Abstract (English)
Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance. This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs, leveraging token-level log-probability differences between the current policy and a reference model. By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity. We further explore the role of reference model quality, demonstrating that stronger reference models significantly improve token selection accuracy and overall optimization effectiveness. Comprehensive experiments on benchmarks such as Arena-Hard and MT-Bench validate the superiority of our Selective-DPO method over standard DPO and distillation-based baselines. Our findings highlight the importance of token-level optimization and reference model selection in advancing preference alignment for LLMs. The code is available at https://github.com/Dongzhijin/SDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。