arXiv:2509.02479cs.LG2025-09被引 143

让大模型多轮调用工具推理更稳定,提升数学解题能力。

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

  • 通过过滤无效推理回合,缓解强化学习训练中的梯度爆炸问题。
  • 在AIME24数学竞赛题上,得分从22.1提升至50.5,超越基线。
  • 无需监督微调,可发现自纠错、交叉验证等复杂推理策略。

大型语言模型通过与外部工具交互可显著增强推理能力,即工具集成推理(TIR)。然而,将TIR扩展至多轮场景时,强化学习常因训练不稳定和性能下降而受阻。我们发现,其主要原因是外部工具反馈引发的分布偏移,导致低概率标记生成,问题随轮次累积,最终引发灾难性梯度范数爆炸。为此,我们提出SimpleTIR,一种即插即用算法,通过识别并剔除无代码块或最终答案的无效回合,阻止有害高梯度参与策略更新,从而稳定学习过程。大量实验表明,SimpleTIR在挑战性数学推理基准上达到领先性能,从Qwen2.5-7B基线出发,使AIME24得分由22.1提升至50.5。此外,由于规避了监督微调限制,模型能自发探索多样且复杂的推理模式,如自纠错和交叉验证。

原文摘要 · Abstract (English)

Large Language Models (LLMs) can significantly improve their reasoning capabilities by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios using Reinforcement Learning (RL) is often hindered by training instability and performance collapse. We identify that such instability is primarily caused by a distributional drift from external tool feedback, leading to the generation of low-probability tokens. This issue compounds over successive turns, causing catastrophic gradient norm explosions that derail the training process. To address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that stabilizes multi-turn TIR training. Its core strategy is to identify and filter out trajectories containing void turns, i.e., turns that yield neither a code block nor a final answer. By removing these problematic trajectories from the policy update, SimpleTIR effectively blocks the harmful, high-magnitude gradients, thus stabilizing the learning dynamics. Extensive experiments show that SimpleTIR achieves state-of-the-art performance on challenging math reasoning benchmarks, notably elevating the AIME24 score from a text-only baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model. Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR encourages the model to discover diverse and sophisticated reasoning patterns, such as self-correction and cross-validation.

强化学习工具调用多轮推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。