arXiv:2603.01223cs.LGcs.CL2026-03被引 6

用参考答案引导生成可学习的推理路径,解决强化学习在难题上奖励稀疏的问题。

Learn Hard Problems During RL with Reference Guided Fine-tuning

  • 用部分参考答案引导模型生成符合自身推理能力的轨迹
  • 在三个数学基准上提升可解题数和强化学习最终性能
  • 适合需要提升复杂推理能力的模型训练场景

数学推理中的强化学习常因奖励稀疏而失效:对于难题,大语言模型无法采样出任何正确轨迹,导致强化学习无法获得有效正反馈。尽管问题常附有人类编写的参考解答(如AoPS数据集),但直接微调这些解答无效,因为模型难以模仿超出自身推理分布的人类证明。本文提出参考引导微调(ReGFT),通过提供部分参考解答,让模型在其自身推理空间内生成推理轨迹,从而合成有意义的正向轨迹用于微调。该方法在三个基准(AIME24、AIME25、BeyondAIME)上均提升了监督学习准确率,加速了DAPO训练,并显著提高强化学习的最终性能上限。结果表明,ReGFT有效缓解了奖励稀疏问题,增强了基于强化学习的数学推理能力。

原文摘要 · Abstract (English)

Reinforcement learning (RL) for mathematical reasoning can suffer from reward sparsity: for challenging problems, LLM fails to sample any correct trajectories, preventing RL from receiving meaningful positive feedback. At the same time, there often exist human-written reference solutions along with the problem (e.g., problems from AoPS), but directly fine-tuning on these solutions offers no benefit because models often cannot imitate human proofs that lie outside their own reasoning distribution. We introduce Reference-Guided Fine-Tuning (ReGFT), a simple and effective method that utilizes human-written reference solutions to synthesize positive trajectories on hard problems and train on them before RL. For each problem, we provide the model with a partial reference solution and let it generate its own reasoning trace, ensuring the resulting trajectories remain in the model's reasoning space while still benefiting from reference guidance. Fine-tuning on these reference-guided trajectories increases the number of solvable problems and produces a checkpoint that receives more positive rewards during RL. Across three benchmarks (AIME24, AIME25, BeyondAIME), ReGFT consistently improves supervised accuracy, accelerates DAPO training, and raises the final performance plateau of RL. Our results show that ReGFT effectively overcomes reward sparsity and unlocks stronger RL-based mathematical reasoning.

强化学习数学推理微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。