arXiv:2509.26553cs.CLcs.PL2025-09被引 11

构建可控的多步工具调用评估框架,揭示大模型状态追踪短板并提出轻量改进方案。

Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling

  • 将工具调用建模为依赖图遍历,通过生成合成任务实现难度精确控制。
  • 依赖深度增加时性能急剧下降,连接型干扰函数最难处理,GPT-5成功率超62.5%至81.3%。
  • 显式重述变量值可显著提升多轮调用稳定性,适合研究大模型工具使用鲁棒性者阅读。

现有工具增强语言模型(TaLMs)基准测试缺乏对任务难度的精细控制,且易受数据污染影响。本文提出 FuncBenchGen,一个统一、无污染的评估框架,通过生成合成的多步工具调用任务来压力测试 TaLMs。核心思想是将工具使用视为隐藏函数依赖有向无环图(DAG)上的遍历过程,模型需推断正确调用顺序以计算目标值。该框架可精确调控任务难度(如图大小、依赖深度、干扰函数),同时避免预训练与测试阶段的数据泄露。实验表明,推理优化模型始终优于通用模型,其中 GPT-5 显著领先。随着依赖深度增加,性能显著下降。此外,共享类型兼容变量的连接型干扰函数尤为棘手。强模型虽常生成语法正确的调用,但会跨步骤传播错误或过期参数值,暴露出大模型在多轮工具使用中状态追踪的脆弱性。基于此,我们提出简单缓解策略:在每一步显式重述前序变量值。令人惊讶的是,这一轻量改动使 GPT-5 成功率从 62.5% 提升至 81.3%,效果显著。

原文摘要 · Abstract (English)

Existing benchmarks for tool-augmented language models (TaLMs) lack fine-grained control over task difficulty and remain vulnerable to data contamination. We present FuncBenchGen, a unified, contamination-free framework that evaluates TaLMs by generating synthetic multi-step tool-use tasks to stress-test TaLMs. The key idea is to cast tool use as traversal over a hidden function-dependency DAG where models must infer the correct sequence of calls to compute a target value. FuncBenchGen allows precise control over task difficulty (e.g., graph size, dependency depth, and distractor functions) while avoiding pretraining/test-time leakage. Our evaluation demonstrates reasoning-optimized models consistently outperform general-purpose models with GPT-5 significantly outperforming other available models. Performance declines sharply as dependency depth increases. Furthermore, connected distractors -- irrelevant functions sharing type-compatible variables with relevant functions -- prove especially difficult to handle. Also, strong models often make syntactically valid function calls but propagate incorrect or stale argument values across steps, revealing brittle state tracking by LLMs in multi-turn tool use. Motivated by this observation, we introduce a simple mitigation strategy that explicitly restates prior variable values to the agent at each step. Surprisingly, this lightweight change yields substantial gains across models. e.g., yielding an improvement in success rate from 62.5% to 81.3% for GPT-5.

大模型评估工具调用状态追踪可控生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。