arXiv:2503.04625cs.CL2025-03EMNLP被引 60

让大模型学会用外部工具自我纠错,提升复杂推理准确率。

START: Self-taught Reasoner with Tools

  • 引入提示触发机制,让模型在推理时自发调用外部工具。
  • 在多个竞赛级评测中超越基础模型,接近顶尖闭源模型表现。
  • 适合需要高精度推理的科研、数学与代码生成场景。

大型推理模型(如OpenAI-o1和DeepSeek-R1)虽在复杂推理任务中表现优异,但因仅依赖内部推理过程,常出现幻觉和效率低下问题。本文提出START(Self-Taught Reasoner with Tools),一种集成外部工具的长链思维推理大模型。通过代码执行,START可完成复杂计算、自我校验、探索多种解法并自调试,从而克服传统模型缺陷。其核心创新在于自学习框架:1)提示触发(Hint-infer)——在推理过程中插入人工设计提示(如“或许用Python是个好主意”),有效激发模型调用外部工具,无需示例数据,亦可作为测试时扩展方法;2)提示拒绝采样微调(Hint-RFT)——结合提示触发与拒绝采样微调,对生成的含工具调用的推理轨迹进行评分、筛选与修正后,再微调模型。基于此框架,我们对QwQ-32B模型进行了优化,得到START。在博士级科学问答(GPQA)、竞赛级数学基准(AMC23、AIME24、AIME25)及竞赛级代码基准(LiveCodeBench)上,分别取得63.6%、95.0%、66.7%、47.1%和47.3%的准确率,显著优于基线模型,性能接近开源最优模型R1-Distill-Qwen-32B和闭源模型o1-Preview。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) like OpenAI-o1 and DeepSeek-R1 have demonstrated remarkable capabilities in complex reasoning tasks through the utilization of long Chain-of-thought (CoT). However, these models often suffer from hallucinations and inefficiencies due to their reliance solely on internal reasoning processes. In this paper, we introduce START (Self-Taught Reasoner with Tools), a novel tool-integrated long CoT reasoning LLM that significantly enhances reasoning capabilities by leveraging external tools. Through code execution, START is capable of performing complex computations, self-checking, exploring diverse methods, and self-debugging, thereby addressing the limitations of LRMs. The core innovation of START lies in its self-learning framework, which comprises two key techniques: 1) Hint-infer: We demonstrate that inserting artificially designed hints (e.g., ``Wait, maybe using Python here is a good idea.'') during the inference process of a LRM effectively stimulates its ability to utilize external tools without the need for any demonstration data. Hint-infer can also serve as a simple and effective sequential test-time scaling method; 2) Hint Rejection Sampling Fine-Tuning (Hint-RFT): Hint-RFT combines Hint-infer and RFT by scoring, filtering, and modifying the reasoning trajectories with tool invocation generated by a LRM via Hint-infer, followed by fine-tuning the LRM. Through this framework, we have fine-tuned the QwQ-32B model to achieve START. On PhD-level science QA (GPQA), competition-level math benchmarks (AMC23, AIME24, AIME25), and the competition-level code benchmark (LiveCodeBench), START achieves accuracy rates of 63.6%, 95.0%, 66.7%, 47.1%, and 47.3%, respectively. It significantly outperforms the base QwQ-32B and achieves performance comparable to the state-of-the-art open-weight model R1-Distill-Qwen-32B and the proprietary model o1-Preview.

推理模型工具调用自我纠错数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。