通过模型置信度筛选关键词元，实现高效精准对齐。
ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
- 依据策略模型置信度识别影响偏好最关键的词元进行优化。
- 在AlpacaEval 2和Arena-Hard上优于传统方法，零额外计算开销。
- 无需辅助模型，轻量且可扩展，适合实际部署场景。
我们提出ConfPO，一种用于大语言模型偏好学习的方法，仅依赖训练策略模型的置信度，识别并优化与偏好相关的关键词元，无需任何辅助模型或额外计算。与以往直接对齐算法（如DPO）不同，后者对所有词元概率进行均匀调整，而ConfPO聚焦于最具影响力的词元，提升对齐质量的同时更高效利用KL散度预算，缓解过优化（奖励黑客）问题。相比近期依赖信用分配模型或AI标注器的词元级方法，其面临可扩展性和可靠性挑战，ConfPO具备简单、轻量、无模型特性。在包含AlpacaEval 2和Arena-Hard在内的多个挑战性对齐基准测试中，实验结果表明，ConfPO在多种LLM上均持续优于统一调整的DAAs，实现更好对齐，且无额外计算开销。
原文摘要 · Abstract (English)
We introduce ConfPO, a method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute. Unlike prior Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token probabilities regardless of their relevance to preference, ConfPO focuses optimization on the most impactful tokens. This targeted approach improves alignment quality while mitigating overoptimization (i.e., reward hacking) by using the KL divergence budget more efficiently. In contrast to recent token-level methods that rely on credit-assignment models or AI annotators, raising concerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs across various LLMs, delivering better alignment with zero additional computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。