arXiv:2505.24480cs.CLcs.AI2025-05被引 17

让AI在解题时自动写代码并执行,提升推理能力与稳定性。

Towards Effective Code-Integrated Reasoning

  • 用强化学习教模型何时、如何调用代码工具
  • 在5个数学基准上显著超越现有模型
  • 揭示代码集成能扩展模型能力边界

本文研究代码集成推理,即模型在需要时生成代码,并通过代码解释器执行获取反馈。为获得此能力,模型需学会有效使用外部代码工具,这由工具增强型强化学习(RL)通过交互式学习实现。尽管有优势,工具增强型RL仍可能面临学习动态不稳定的挑战。为此,我们提出系统性方法,提升训练效率与稳定性:设计增强训练策略,在探索与稳定间取得平衡,逐步构建工具使用能力并提升推理性能。在五个主流数学推理基准上的大量实验表明,本模型显著优于多个竞争基线。此外,我们深入分析了代码集成推理的机制与效果,揭示若干关键洞察,如模型能力边界的拓展以及通过代码集成实现推理效率的同步提升。所有数据和代码可于 https://github.com/RUCAIBox/CIR 获取。

原文摘要 · Abstract (English)

In this paper, we investigate code-integrated reasoning, where models generate code when necessary and integrate feedback by executing it through a code interpreter. To acquire this capability, models must learn when and how to use external code tools effectively, which is supported by tool-augmented reinforcement learning (RL) through interactive learning. Despite its benefits, tool-augmented RL can still suffer from potential instability in the learning dynamics. In light of this challenge, we present a systematic approach to improving the training effectiveness and stability of tool-augmented RL for code-integrated reasoning. Specifically, we develop enhanced training strategies that balance exploration and stability, progressively building tool-use capabilities while improving reasoning performance. Through extensive experiments on five mainstream mathematical reasoning benchmarks, our model demonstrates significant performance improvements over multiple competitive baselines. Furthermore, we conduct an in-depth analysis of the mechanism and effect of code-integrated reasoning, revealing several key insights, such as the extension of model's capability boundaries and the simultaneous improvement of reasoning efficiency through code integration. All data and code for reproducing this work are available at: https://github.com/RUCAIBox/CIR.

代码推理强化学习数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。