FocalPO改进偏好优化，专注提升正确排序能力。
FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
- 通过动态降低错误排序样本权重，聚焦训练正确排序对
- 在Alpaca Eval 2.0上优于DPO及其变体，无需调参
- 适合追求高效对齐的LLM训练者，尤其适配Mistral和Llama模型
高效偏好优化算法如直接偏好优化（DPO）已成为对齐大语言模型（LLMs）与人类偏好的主流方法。这类算法隐式将LLM视为奖励模型，专注于纠正错误排序的偏好对。然而，近期研究发现，尽管DPO梯度强调错误排序对，其训练却极少改善这些样本。本文提出FocalPO，一种改进的DPO方法：反向降低错误排序对的权重，优先提升模型对已能正确排序对的理解。受视觉任务中焦点损失（Focal Loss）启发，FocalPO通过引入调制因子动态调节DPO损失。实验表明，使用Mistral-Base-7B和Llama-3-Instruct-8B，在Alpaca Eval 2.0等基准上，FocalPO超越DPO及其变体，且超参数固定不变。此外，我们实证揭示了FocalPO对正确与错误样本组训练的影响，进一步验证其有效性。
原文摘要 · Abstract (English)
Efficient preference optimization algorithms such as Direct Preference Optimization (DPO) have become a popular approach in aligning large language models (LLMs) with human preferences. These algorithms implicitly treat the LLM as a reward model, and focus on training it to correct misranked preference pairs. However, recent work~\citep{chen2024preference} empirically finds that DPO training \textit{rarely improves these misranked preference pairs}, despite its gradient emphasizing on these cases. We introduce FocalPO, a DPO variant that instead \textit{down-weighs} misranked preference pairs and prioritizes enhancing the model's understanding of pairs that it can already rank correctly. Inspired by Focal Loss used in vision tasks, FocalPO achieves this by adding a modulating factor to dynamically scale DPO loss. Our experiment demonstrates that FocalPO surpasses DPO and its variants on popular benchmarks like Alpaca Eval 2.0 using Mistral-Base-7B and Llama-3-Instruct-8B, with the introduced hyperparameter fixed. Additionally, we empirically reveals how FocalPO affects training on correct and incorrect sample groups, further underscoring its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。