arXiv:2608.09939cs.HCcs.AI2026-08

用虚拟角色模拟真实任务,检验AI聊天机器人能否帮用户达成目标。

How to Dogfood Your AI Chat Agent: A Three-Layer Evaluation Framework with Goal-Directed NPC Simulation

  • 分三层测试:基础问答、随机对话、目标驱动的虚拟角色模拟。
  • 虚拟角色模拟实现77%任务完成率,成本仅0.17美元/次,比人工便宜6272倍。
  • 适合在生产环境中持续集成的AI聊天系统质量评估,支持自动发布决策。

部署大型语言模型聊天代理的生产团队面临一个质量保障空白:现有评估工具要么测试单轮回复,要么模拟社交互动,但均无法系统验证真实用户是否能通过多轮对话达成目标。本文提出一种三层自测框架,结合标准问题库测试(第1层)、随机多轮对话评估(第2层)和目标导向的非玩家角色(NPC)模拟器(第3层),后者包含五类结构化目标与十类失败分类。在为期三个月的纵向案例研究中(共257次评估运行,涵盖108个场景的NPC套件),发现三层评估产生互补的回归信号:同一轮内响应质量相关性弱(斯皮尔曼等级相关系数-0.15至0.14),跨时间序列为负相关(低至-0.46),证实标准正确性无法预测目标导向对话成功。该NPC模拟器以0.17美元/次的成本实现77%的目标达成率,比人工评估便宜6272倍,支持每日CI/CD集成及自动化发布(PROMOTE/HOLD/ROLLBACK)决策。论文开源全部提示模板、失败分类体系与Python可复现指南,供其他团队直接采用。

原文摘要 · Abstract (English)

Production teams deploying LLM chat agents face a specific quality assurance gap: existing evaluation tools test individual responses or simulate social interactions, but none systematically verify whether real users can achieve their goals through multi-turn conversation. We introduce a three-layer dogfooding framework that bridges this gap by combining canonical question-bank testing (Layer 1), random-walk multi-turn evaluation (Layer 2), and a goal-directed NPC (Non-Player Character) simulator with five structured goal types and a ten-category failure taxonomy (Layer 3). In a longitudinal case study on a production multi-agent system over roughly three months (257 evaluation runs; a 108-scenario NPC suite), we find that the three layers produce complementary regression signals: cross-layer correlation for response quality is weak within a synchronized run (Spearman rho between -0.15 and 0.14) and negative across the longitudinal series (rho down to -0.46), confirming that canonical correctness does not predict goal-directed conversation success. The NPC simulator achieves 77 percent goal achievement at 0.17 dollars per run (6,272x cheaper than human evaluation), enabling daily CI/CD integration with automated PROMOTE/HOLD/ROLLBACK release decisions. We release full prompt templates, the failure taxonomy, and a Python-first replicability guide so that other teams can adopt the framework for their own LLM chat agents.

AI评估聊天机器人自动化测试目标导向

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。