arXiv:2605.27788cs.LGcs.CL2026-05被引 2

让大模型学会在何时该用工具,提升推理准确率。

Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

  • 基于模型自生成轨迹,分段分配奖励以判断何时需调用工具。
  • 在5个基准上提升准确率6.7~9.7点,小模型收益更显著。
  • 能减少无效调用,适合资源有限的小规模模型使用。

人类知道何时需要求助——比如347×28要用计算器,而2+2则无需。语言模型却缺乏这种判断力。传统提示方法虽可指导工具调用,但无法教会模型识别自身知识边界。强化学习方法对整个轨迹给予单一奖励,无法定位具体哪次工具调用真正有效,也无法惩罚冗余调用。本文提出CARL(Competence-Aware Reinforcement Learning),通过在模型自生成的轨迹中,按自然工具调用边界(如代码块分隔符、上下文切换)分解每条轨迹,仅用二元结果即可独立分配各段信用,无需外部标注或逐步标注。由此,错误调用、提取错误和冗余调用均获得恰当的奖励信号。训练后的评估器能准确捕捉模型领域能力:在7B模型上,对可参数化解决与依赖工具的问题分类的AUC达0.93。在涵盖算术、多跳事实问答和财务表格数值推理的五个基准上,相比最优强化学习基线,CARL在7B模型上提升精确匹配准确率6.7点,在3B模型上提升9.7点,其中在Musique上的最大提升为+8.3(7B)和+9.0(3B)。模型在可参数化回答的问题上工具调用减少53%,同时准确率仍高出约10点。小模型受益更明显:3B模型的增益是7B模型的1.4倍,表明‘何时提问’的判断对小模型尤为重要。

原文摘要 · Abstract (English)

Humans know when to reach for help e.g. $347 \times 28$ warrants a calculator while $2+2$ does not. Language models do not. Prompt-based approaches can instruct a model when to invoke tools, but this scaffolding does not teach it to recognize the boundary of its own knowledge. RL approaches that assign a single outcome reward to the whole trajectory fare no better: trajectory-level credit cannot isolate which tool call in a successful episode actually helped, nor penalize unnecessary calls. We propose \textbf{CARL} (\textbf{C}ompetence-\textbf{A}ware \textbf{R}einforcement \textbf{L}earning), which trains a critic on the model's own rollouts to learn where parametric knowledge suffices and where it needs external help. By decomposing each rollout at natural tool-use boundaries (e.g., code fence delimiters and context block transitions), CARL assigns independent credit to each segment from a single binary outcome, without external judges or step-level annotations. As a result, erroneous tool calls, incorrect extractions, and unnecessary calls each receive appropriately signed advantages. The trained critic captures the model's domain competence: it separates parametrically solvable from tool-dependent questions with AUC 0.93 at 7B. On five benchmarks spanning arithmetic, multi-hop factual QA, and numerical reasoning over financial tables, CARL improves exact-match accuracy by 6.7 points at 7B and 9.7 points at 3B over the best RL baseline, with the largest gain (+8.3 EM at 7B, +9.0 EM at 3B) on Musique. The model issues 53\% fewer tool calls on parametrically answerable questions while remaining ${\sim}10$ EM points more accurate on them. Gains are largest at small scale: the 3B improvement is $1.4\times$ the 7B improvement, suggesting that knowing when to ask disproportionately benefits models with smaller parametric memory.

大模型工具调用强化学习推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。