用工具交互替代自然语言思考，让小模型也能高效推理。
Replacing thinking with tool usage enables reasoning in small language models
- 将思考过程转化为与状态化工具的多轮交互，用领域专用语言控制工具。
- 30亿参数的小模型在修复Python代码任务上达到媲美大模型的表现。
- 适合需要高效推理的小模型应用，如边缘设备或实时系统。
近期研究提出一种新范式：通过在推理和训练阶段同时扩展计算量，使大语言模型在推理时以自然语言形式生成‘思考’过程。本文提出改用状态化工具的多轮交互来替代自然语言思考。每轮中，工具的新状态被加入模型上下文，模型则生成用于控制工具的领域专用语言（DSL）指令。我们在修复故障Python代码的任务上进行评估，结果表明该受限设置能加快经验采样速度并提供更密集的奖励信号，使最大仅30亿参数的模型也能有效利用额外计算资源完成任务。
原文摘要 · Abstract (English)
Recent advances have established a new machine learning paradigm based on scaling up compute at inference time as well as at training time. In that line of work, a combination of Supervised Fine-Tuning (SFT) on synthetic demonstrations and Reinforcement Learning with Verifiable Rewards (RLVR) is used for training Large Language Models to expend extra compute during inference in the form of "thoughts" expressed in natural language. In this paper, we propose to instead format these tokens as a multi-turn interaction trace with a stateful tool. At each turn, the new state of the tool is appended to the context of the model, whose job is to generate the tokens necessary to control the tool via a custom DSL. We benchmark this approach on the problem of repairing malfunctioning Python code, and show that this constrained setup allows for faster sampling of experience and a denser reward signal, allowing even models of size up to 3B parameters to learn how to proficiently expend additional compute on the task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。