提出工具调用鲁棒性评估基准,提升真实场景下AI Agent的稳定性。
When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents

- 构建四类扰动基准,模拟真实部署中的输入、动作、奖励和状态错误。
- 大模型在动态扰动下准确率下降超30%,规模无法自动弥补缺陷。
- 通过域随机强化学习训练,显著提升对未知运行错误的容错能力。
工具调用语言智能体在评估中假设输入干净、工具注册清晰、接口可靠;但真实部署中用户误输入会引发工具名幻觉,配置错误导致代理无限停滞,跨服务器工具名重复可使SDK冻结。本文将这些失败视为工具使用部分可观测马尔可夫决策过程(POMDP)中的仿真到现实差距,噪声通过观测、动作空间、奖励相关元数据或转移动态进入系统。提出RobustBench-TC基准,包含22种按四类POMDP组件组织的扰动类型,均基于已验证的GitHub问题或工具调用失败记录。在21个从1.5B到32B参数的模型(含闭源o4-mini)上测试发现:观测扰动仅使准确率下降<5%,而奖励相关与转移扰动分别导致约40%和30%下降;模型规模无法弥合此差距。随后提出ToolRL-DR,一种域随机强化学习训练方案,在静态编码的三类POMDP组件扰动轨迹上训练工具调用智能体。以3B主干模型为例,ToolRL-DR-Full保留约75%清洁准确率,综合扰动准确率接近开源14B函数调用基线,且显著缩小与o4-mini的差距。其在训练中未接触转移扰动,却关闭了约27%的转移差距,表明对抗性静态输入上的强化学习能诱导更持久的重试策略,并泛化至未见运行故障。数据集、代码及排行榜均已公开。
原文摘要 · Abstract (English)
Tool-use language agents are evaluated on benchmarks that assume clean inputs, unambiguous tool registries, and reliable APIs. Real deployments violate all these assumptions: user typos propagate into hallucinated tool names, a misconfigured request timeout can stall an agent indefinitely, and duplicate tool names across servers can freeze an SDK. We study these failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP), where deployment noise enters through the observation, action space, reward-relevant metadata, or transition dynamics. We introduce RobustBench-TC, a benchmark with 22 perturbation types organized by these four POMDP components, each grounded in a verified GitHub issue or documented tool-calling failure. Across 21 models from 1.5B to 32B parameters (including the closed-source o4-mini), the robustness profile is sharply uneven: observation perturbations reduce accuracy by less than 5%, while reward-relevant and transition perturbations reduce accuracy by roughly 40% and 30%, respectively; scale alone does not close these gaps. We then propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories spanning the three statically encodable POMDP components. On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate perturbed accuracy comparable to open-source 14B function-calling baselines while substantially narrowing the gap to o4-mini. It closes approximately 27% of the Transition gap despite never seeing transition perturbations in training, suggesting that RL on adversarial static tool-use inputs induces a more persistent retry policy that transfers to unseen runtime failures. The dataset, code and benchmark leaderboard are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。