通过主动选择信息量大的负样本，提升偏好优化效率与模型对齐效果。
MASS-DPO: Multi-negative Active Sample Selection for Direct Policy Optimization
- 基于普拉克特-卢瑟模型，用费雪信息构造目标函数筛选互补负样本。
- 仅用少量负样本即达到全池训练的性能，召回率与NDCG显著提升。
- 适合需要高效微调的大模型，尤其在推荐和多选题任务中表现优异。
在普拉克特-卢瑟（Plackett--Luce）模型下，多负样本偏好优化通过对比一个优选响应与多个拒选响应来扩展直接偏好优化（DPO）。然而，在大规模负样本池上进行优化代价高昂，且许多候选样本因对策略更新影响相似而贡献冗余梯度。本文提出MASS-DPO，一种针对PL模型的主动样本选择方法，其基于费雪信息构建目标函数，从每个提示中选取紧凑且信息丰富的负样本子集。该方法的对数行列式目标函数能选出在策略更新中提供互补信息的负样本，实现信息保留的同时降低冗余。实践中，该方法更倾向于选择梯度方向差异大的负样本，减少近似重复样本带来的冗余信号，同时保留最有价值的训练信息。在涵盖推荐与多项选择问答的四个基准上，跨三个模型家族的实验表明，MASS-DPO在准确率上持续优于或匹配现有方法，提升了召回率（Recall）与NDCG，改善了基于边际的优化动态，并以极少数负样本实现了更强的对齐效果。
原文摘要 · Abstract (English)
Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。