让评测环境可编程演进,真实模拟工具调用智能体在动态变化中的表现。
The World Won't Stay Still: Programmable Evolution for Agent Benchmarks
- 用带类型的关系图统一表示环境数据、工具与结构,实现演化可编程。
- 支持自动构建演进后的可执行环境,且能生成任务沙箱用于测试。
- 适用于电商与机票预订场景,可诊断智能体在环境变化下的适应能力。
大语言模型驱动的工具调用智能体通过多轮交互与环境、数据和工具协作完成用户请求。然而,现有评测大多基于静态环境接口,固定数据模式与工具集,难以评估智能体在环境持续演化时的表现——例如功能新增、重组或废弃。本文将结构化环境演化视为智能体评测框架构建问题,提出 ProEvolve:一种基于图的可编程演化框架。其核心是一个带类型的关联图,统一表达环境中的数据、工具与模式。在此形式下,功能增删改均以图变换形式表达,并在工具、模式与数据访问间一致传播更新。ProEvolve 支持(1)通过显式图变换自动生成演化后的可执行环境;(2)基于子图采样与实例化构建任务沙箱。我们在电商与机票预订两个领域验证了其在质量、实现有效性及失效模式上的表现。最后,利用生成的基准进行下游诊断,研究代表性智能体在结构化环境演化下的行为变化。
原文摘要 · Abstract (English)
LLM-powered tool-calling agents fulfill user requests by interacting with environments, querying data, and invoking tools in a multi-turn process. Yet, most existing benchmarks evaluate these systems under static environment interfaces, with fixed schemas and toolsets, making it difficult to assess how agents behave as environments evolves -- when capabilities are added, reorganized, or deprecated across successive environment versions. In this paper, we study structured environment evolution as a benchmark-construction problem for tool-calling agents. We propose ProEvolve, a graph-based framework that makes environment evolution programmable. At its core, a typed relational graph provides a unified, explicit representation of the environment - data, tools, and schema. Under this formalism, adding, removing, or modifying capabilities are expressed as graph transformations that coherently propagate updates across tools, schemas, and data access. Building on this, ProEvolve supports (1) automatic generation of evolved executable environments through explicit graph transformations, and (2) graph-grounded construction of task sandboxes via subgraph sampling and instantiation. We validate ProEvolve in two tool-calling domains, e-commerce and airline booking, in terms of quality, implementation validity, and failure modes. Finally, we use the generated benchmark as a downstream diagnostic to study how representative agents behave under structured environment evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。