通过每步奖励提升数学推理模型性能，无需人工标注错误步骤。
Full-Step-DPO: Self-Supervised Preference Optimization with Step-wise Rewards for Mathematical Reasoning
- 用自监督模型自动给推理每一步打分，生成步级奖励。
- 在多个数学推理数据集上超越现有最佳方法，提升显著。
- 适合需要强逻辑推理能力的模型优化场景。
直接偏好优化（DPO）在长链数学推理任务中表现不佳。现有方法如 Step-DPO 仅关注推理链中的首个错误步骤，忽略其余步骤，且依赖人工或 GPT-4 标注错误。为此，我们提出 Full-Step-DPO，一种专为数学推理设计的新 DPO 框架。该方法不局限于首个错误步骤，而是利用整个推理链的步级奖励。通过训练一个自监督过程奖励模型，自动为每一步生成评分，避免对外部信号的依赖。同时引入新型步级 DPO 损失函数，根据步级奖励动态调整梯度。实验证明，在多种基础语言模型及域内、域外数学推理基准测试中，Full-Step-DPO 均显著优于当前最优基线。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) often struggles with long-chain mathematical reasoning. Existing approaches, such as Step-DPO, typically improve this by focusing on the first erroneous step in the reasoning chain. However, they overlook all other steps and rely heavily on humans or GPT-4 to identify erroneous steps. To address these issues, we propose Full-Step-DPO, a novel DPO framework tailored for mathematical reasoning. Instead of optimizing only the first erroneous step, it leverages step-wise rewards from the entire reasoning chain. This is achieved by training a self-supervised process reward model, which automatically scores each step, providing rewards while avoiding reliance on external signals. Furthermore, we introduce a novel step-wise DPO loss, which dynamically updates gradients based on these step-wise rewards. This endows stronger reasoning capabilities to language models. Extensive evaluations on both in-domain and out-of-domain mathematical reasoning benchmarks across various base language models, demonstrate that Full-Step-DPO achieves superior performance compared to state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。