用迭代DPO提升大模型推理能力，效果接近强化学习但更省算力。
Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
- 通过多轮偏好学习，让生成器和评分模型相互优化。
- 单轮粗过滤DPO就能显著提升强基座模型的数学推理能力。
- 仅用可验证奖励即达强化学习水平，适合算力有限场景。
近期大语言模型后训练方法的发展表明，强化学习（RL）在提升推理能力方面至关重要。然而，基于RL的方法计算成本高昂，促使研究者关注替代范式如直接偏好优化（DPO）。本文系统考察了DPO在实现大模型自我改进方面的有效性，提出一种迭代增强框架，使生成器与奖励模型通过多轮DPO在线交互实现互惠提升。研究发现，单轮粗过滤的DPO即可显著增强强基座模型的数学推理表现；结合简单可验证奖励，所提模型DPO-VP在推理性能上达到与强化学习相当的水平，同时大幅降低计算开销。结果表明，DPO是可扩展且低成本的替代方案，为资源受限环境下的大模型推理增强提供了实用路径。
原文摘要 · Abstract (English)
Recent advancements in post-training methodologies for large language models (LLMs) have highlighted reinforcement learning (RL) as a critical component for enhancing reasoning. However, the substantial computational costs associated with RL-based approaches have led to growing interest in alternative paradigms, such as Direct Preference Optimization (DPO). In this study, we investigate the effectiveness of DPO in facilitating self-improvement for LLMs through iterative preference-based learning. We demonstrate that a single round of DPO with coarse filtering significantly enhances mathematical reasoning performance, particularly for strong base model. Furthermore, we design an iterative enhancement framework for both the generator and the reward model (RM), enabling their mutual improvement through online interaction across multiple rounds of DPO. Finally, with simple verifiable rewards, our model DPO-VP achieves RL-level performance with significantly lower computational overhead. These findings highlight DPO as a scalable and cost-effective alternative to RL, offering a practical solution for enhancing LLM reasoning in resource-constrained situations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。