用新指标动态调整工具列表长度,让大模型选得更快更准。
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer

- 提出机会校正指标BoR,评估不同工具列表长度的有效性。
- 自适应策略平均只展示7个工具,却达到90.3%的覆盖率达50个工具水平。
- 适合需要高效工具选择的智能体系统,尤其在复杂任务中表现突出。
在LLM智能体使用工具前,检索系统需决定向其展示哪些候选工具。展示数量过少可能遗漏正确工具,过多则导致模型难以选择。现有系统多采用固定列表长度,但缺乏评估标准。本文将工具列表长度作为评估对象,引入比特-随机比(BoR)这一机会校正指标,判断在特定深度的成功率是否优于随机选择。我们在三个工具选择基准上验证了BoR,涵盖20至3,251个工具的注册表,结合多种评分器。进一步将该原理用于强化学习奖励,训练智能体按查询动态选择列表长度。实验显示,在BFCL(370工具)上,该策略平均仅展示7个工具,覆盖率90.3%,接近固定展示50个工具的90.8%。在ToolBench(3,251工具)上,固定5个工具的总覆盖率为64.7%(高于固定50的61.9%),但在难例(正确工具排名6-20)上无法找到,而本方法在这些案例中实现了16.7%的覆盖率。下游测试表明,自适应短列表显著提升选对率:在Claude Sonnet 4.6上达93.1%(固定5工具为87.1%),中等难度任务下差距扩大至76.8% vs 60.9%。
原文摘要 · Abstract (English)
Before an LLM agent can use a tool, a retrieval system must decide which candidate tools to show to the agent. How long should that shortlist be? Show too many tools and the model struggles to choose. Show too few and the correct tool may not appear. Most systems apply a fixed shortlist size to every query, but no standard metric exists to evaluate whether that size was appropriate. We treat the number of tools shown to an LLM agent as the object of evaluation and we apply Bits-over-Random (BoR), a chance-corrected metric that asks whether success at a given depth is better than what random selection would achieve at that same depth. We evaluate BoR across three tool-selection benchmarks, multiple scorers, and registries ranging from 20 to 3,251 tools. We then turn the same principle into a reinforcement learning (RL) reward for choosing tool shortlist depth per query. The RL agent is deliberately simple, serving as a probe of the metric rather than a proposed system. As the shortlist grows, random chance of including the correct tool rises, so the reward naturally decreases, reducing the need for an engineered depth penalty. On BFCL (370 tools), the learned policy nearly matches the coverage of showing 50 tools ($90.3\%$ vs $90.8\%$) while presenting only 7 on average. On ToolBench (3,251 tools), a fixed shortlist of 5 tools achieves higher aggregate coverage ($64.7\%$ vs $61.9\%$) but finds nothing on hard queries (correct tool ranked 6th-20th). The BoR agent finds $16.7\%$ on those same queries by searching deeper. Downstream validation with Claude Sonnet 4.6 indicates that shorter adaptive lists also improve the LLM's ability to select the right tool: $93.1\%$ versus $87.1\%$ when always shown 5 tools, widening to $76.8\%$ vs $60.9\%$ on medium-difficulty queries where the correct tool is present but not ranked first.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。