用连续得分优化大模型推理，提升策略选择和执行能力。
Continuous-Utility Direct Preference Optimization
- 用连续分数替代二值偏好，捕捉推理过程的细微质量
- 策略选择准确率从35-46%提升至68-78%，推理得分最高增6.6点
- 适合需要精细推理与策略优化的研究者
大型语言模型的推理常被视为单一能力，依赖二值偏好监督，难以捕捉部分进展或细粒度推理质量。我们提出连续效用直接偏好优化（CU-DPO），通过将二值标签替换为连续得分，使模型学习一组基于提示的认知策略。理论证明，使用K个策略可使样本复杂度降低至Theta(K log K)；DPO收敛于熵正则化效用最大化策略。为此，我们设计两阶段训练：(i) 策略选择，通过最佳对所有比较优化模型选最优策略；(ii) 执行精炼，使用分层边距对训练模型正确执行所选策略。在数学推理基准测试中，CU-DPO将策略选择准确率从35-46%提升至68-78%，在分布内数据集上推理得分提升达6.6点，并实现有效的分布外迁移。
原文摘要 · Abstract (English)
Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce Continuous Utility Direct Preference Optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences, and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage training pipeline: (i) strategy selection, which optimizes the model to choose the best strategy for a given problem via best-vs-all comparisons, and (ii) execution refinement, which trains the model to correctly execute the selected strategy using margin-stratified pairs. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46 percent to 68-78 percent across seven base models, yielding consistent downstream reasoning gains of up to 6.6 points on in-distribution datasets with effective transfer to out-of-distribution tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。