VISTA通过混合交互模拟,提升智能体评估的全面性与真实性。
VISTA: A Versatile Interactive User Simulation Toolkit for Agent Evaluation

- 构建六项指标,量化模拟交互的逼真度与覆盖能力
- 融合界面与API操作,支持更真实的多步骤用户行为模拟
- 适用于电商与客服等复杂交互场景,适合评估智能体真实表现
评估仍是交互式智能体开发中的关键瓶颈。现有方法多依赖静态基准,难以捕捉智能体行为的动态性与多步特性,也难暴露其真实失败模式。虽然基于用户模拟的评估有潜力,但现有框架存在两大局限:一是缺乏对模拟交互质量与覆盖度的有效度量手段,难判断是否充分探测了智能体的能力与缺陷;二是多数框架仅支持界面操作或API调用之一,无法建模完整的现实用户行为。为此,我们提出VISTA——一个面向智能体评估的多功能交互用户模拟工具包。该工具包包含六项指标,用于衡量模拟交互的逼真度、能力覆盖范围和交互有效性。同时,我们设计了一个融合界面与API操作的混合用户模拟器,可在多样化交互环境中实现更真实、全面的评估。我们在电商购物与教育客服场景中验证了VISTA,结果表明其相比现有方法能生成更真实、更全面的评估结果。
原文摘要 · Abstract (English)
Evaluation remains a critical bottleneck for interactive agent development. Existing evaluation methods often rely on static benchmarks, which fail to capture the dynamic, multi-step nature of agentic behavior and struggle to expose meaningful failure modes. While user-simulation-based evaluation offers a promising alternative, existing simulation frameworks suffer from two major limitations. First, they provide limited mechanisms for evaluating the quality and comprehensiveness of simulated interactions, making it difficult to assess whether a simulator sufficiently explores an agent's capabilities and failure modes. Second, most frameworks are restricted to either UI-only actions or API-only actions, limiting their ability to model the full range of realistic user behaviors. To address these limitations, we propose VISTA, a Versatile Interactive user Simulation Toolkit for Agent evaluation. Our toolkit includes a suite of six metrics for measuring the realism, capability coverage, and interaction effectiveness of simulated interactions. In addition, we develop a hybrid user simulator that integrates both UI-based interactions and API-based interactions, enabling more realistic and comprehensive evaluation across diverse interactive environments. We evaluate VISTA in e-commerce shopping and education customer service settings and demonstrate that it produces more realistic and comprehensive evaluations than existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。