arXiv:2505.21668cs.AIcs.CL2025-05被引 6

让大模型通过代码推理完成多样任务,准确率提升超70%。

R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning

  • 分阶段训练,优先学提升潜力大的任务
  • 14B模型在37项测试中准确率达72.4%
  • 自动生成代码并自发检查结果,适合复杂推理场景

现有方法缺乏对大语言模型(LLMs)如何有效使用代码解释器的实用指导。我们提出R1-Code-Interpreter,基于文本模型通过多轮监督微调(SFT)和强化学习(RL)训练,实现跨任务的逐步推理与多轮代码生成。针对任务多样性高、有效样本少的问题,我们构建了144个涵盖不同领域的推理与规划任务,并设计分阶段课程学习策略,按提升潜力排序样本,优先训练高潜力任务。该方法使平均强化学习收益从+3.4%提升至+9.3%(在Qwen-2.5系列模型上)。最终模型R1-CI-14B在37个测试任务上的平均准确率从44.1%提升至72.4%,超越纯文本GPT-4o(58.6%)和带代码解释器的GPT-4o(70.9%)。值得注意的是,模型还展现出自动生成代码并自我验证的涌现行为。数据集、代码与模型已公开于https://github.com/yongchao98/R1-Code-Interpreter和https://huggingface.co/yongchao98。

原文摘要 · Abstract (English)

Practical guidance on training Large Language Models (LLMs) to leverage Code Interpreter across diverse tasks remains lacking. We present R1-Code-Interpreter, an extension of a text-only LLM trained via multi-turn supervised fine-tuning (SFT) and reinforcement learning (RL) to autonomously generate multiple code queries during step-by-step reasoning. Unlike prior RL + tool-use efforts focused on narrow domains such as math or retrieval, we curate 144 diverse reasoning and planning tasks and show that training a general-purpose Code Interpreter across them presents significant challenges due to task heterogeneity and scarcity of effective samples. To address this, we introduce a multi-stage curriculum learning approach that partitions training samples by measured improvement potential. The RL training prioritizes samples with higher potential and gradually shifts to lower-potential ones, increasing the average RL gains from merely +3.4% to +9.3% across Qwen-2.5 models (3/7/14B). Our final model, R1-CI-14B, improves average accuracy on the 37 test tasks from 44.1% to 72.4%, outperforming text-only GPT-4o (58.6%) and GPT-4o with Code Interpreter (70.9%). Notably, R1-CI-14B also exhibits emergent self-checking behavior through code generation. Datasets, Codes, and Models are available at https://github.com/yongchao98/R1-Code-Interpreter and https://huggingface.co/yongchao98.

代码推理强化学习大模型自主决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。