针对大模型排名任务，提出基于前K项的优化方法，更贴近真实场景。
K-order Ranking Preference Optimization for Large Language Models
- 扩展DPO的普莱特-卢克模型，支持对前K项排序进行优化
- 在多个数据集上实现更高样本效率与抗噪声能力
- 可动态调整每条查询的K值，适合不同需求的排名场景
为使大语言模型适应排名任务，现有列表级方法（如列表级直接偏好优化，DPO）侧重于优化部分或全序列表的一致性以提升排名能力。然而我们认为，针对实际应用，优化前K项排序一致性更为合适。原因有二：其一，用户通常只关注前K个结果，因此前K项更重要；其二，尾部项目往往缺乏精确反馈，使得前K项排序更具可靠性。基于此，我们提出K阶排名偏好优化（KPO），通过将DPO的普莱特-卢克模型扩展至支持前K项排名。此外，考虑到不同查询中重要项数量可变，我们进一步设计了动态确定各样本适宜K值的方法，并引入课程学习策略以提升训练效率。大量实验表明，KPO在多个基准上均表现出显著有效性，尤其具备高样本效率和强噪声鲁棒性。代码已开源：https://github.com/Lanyu0303/KPO。
原文摘要 · Abstract (English)
To adapt large language models (LLMs) to ranking tasks, existing list-wise methods, represented by list-wise Direct Preference Optimization (DPO), focus on optimizing partial-order or full-order list ranking consistency for LLMs to enhance their ranking abilities. However, we argue that optimizing top-K ranking consistency could be more appropriate for real-world applications. There are two main reasons: (1) users are typically concerned with only the top-K results, making top-K ranking more important, and (2) tail items often lack precise feedback, making top-K ranking more reliable. Based on this, we propose K-order Ranking Preference Optimization (KPO) by extending the DPO's Plackett-Luce model to accommodate top-K rankings. Additionally, recognizing that the number of important items can vary across queries, we extend KPO to dynamically determine appropriate K for different samples and introduce a curriculum learning strategy to boost training efficiency. Extensive experiments demonstrate the effectiveness of KPO, highlighting its high sample efficiency and robustness to noise. The code is available at https://github.com/Lanyu0303/KPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。