用分支模拟提升大模型客服评估效率，发现更多隐藏缺陷。
Efficient Agent Evaluation via Diversity-Guided User Simulation
- 在关键节点保存对话状态，复用共享前缀减少重复计算。
- 通过多样用户回复分支探索，失败发现率提升30%以上。
- 适合需要高效覆盖罕见交互场景的模型评测团队。
大型语言模型（LLMs）越来越多地被用作面向客户的智能代理，但其可靠性评估面临挑战，因交互具有随机性且多轮复杂。现有评估依赖线性蒙特卡洛滚动生成完整对话来估计成功率，但计算效率低，重复生成相同早期对话片段，且难以发现由罕见用户行为引发的深层失败。本文提出DIVERT（Diversity-Induced Evaluation via Branching of Trajectories），一种基于快照、覆盖引导的用户模拟框架，实现对代理-用户交互的系统性探索。该框架在关键决策点捕获完整的代理-环境状态，并从这些快照恢复执行，从而复用共享对话前缀，减少冗余计算。每个交汇点通过针对性的多样性诱导用户回复进行分支，实现对不同交互路径的定向探索。通过聚焦语义多样且未充分探索的轨迹，DIVERT提升了评估效率与覆盖范围。实验证明，相比标准线性滚动协议，其单位令牌内发现的失败数量增加超过30%，同时扩展了可识别失败的任务集合。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed as customer-facing agents, yet evaluating their reliability remains challenging due to stochastic, multi-turn interactions. Current evaluation protocols rely on linear Monte Carlo rollouts of complete agent-user conversations to estimate success. However, this approach is computationally inefficient, repeatedly regenerating identical early prefixes, and often fails to uncover deep failure modes that arise from rare user behaviors. We introduce DIVERT (Diversity-Induced Evaluation via Branching of Trajectories), an efficient, snapshot-based, coverage-guided user simulation framework for systematic exploration of agent-user interactions. DIVERT captures the full agent-environment state at critical decision points and resumes execution from these snapshots, enabling reuse of shared conversation prefixes and reducing redundant computation. From each junction, the framework branches using targeted, diversity-inducing user responses, allowing directed exploration of alternative interaction paths. By focusing evaluation on semantically diverse and underexplored trajectories, DIVERT improves both efficiency and coverage. Empirical results show that it discovers more failures per token compared to standard linear rollout protocols, while expanding the set of tasks on which failures are identified.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。