用智能体自动测试对话智能体，发现更多真实缺陷。
Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents
- 构建元智能体，融合多策略生成对抗性测试用例。
- 比人工标注更快发现多样且严重的失败，20-30分钟完成。
- 适合开发人员快速定位智能体漏洞，支持开源复现。
大型语言模型智能体越来越多地用于规划、检索和工具调用，但评估仍依赖静态基准和小规模人工测试。我们提出代理测试代理（ATA），一个元智能体，结合静态代码分析、设计者问询、文献挖掘和基于角色的对抗性测试生成，并通过评判反馈动态调整测试难度。每个对话使用大模型作为裁判（LAAJ）评分，并引导后续测试聚焦于智能体最薄弱环节。在旅行规划和维基撰写任务中，ATA发现的故障比专家标注更丰富且严重程度相当，耗时仅20至30分钟，而人工标注需十名标注员数日完成。去除代码分析和网络搜索会导致测试方差增大和偏差加剧，凸显基于证据的测试生成价值。ATA输出量化指标与定性缺陷报告供开发者使用。我们已公开完整方法与开源实现：https://github.com/KhalilMrini/Agent-Testing-Agent。
原文摘要 · Abstract (English)
LLM agents are increasingly deployed to plan, retrieve, and write with tools, yet evaluation still leans on static benchmarks and small human studies. We present the Agent-Testing Agent (ATA), a meta-agent that combines static code analysis, designer interrogation, literature mining, and persona-driven adversarial test generation whose difficulty adapts via judge feedback. Each dialogue is scored with an LLM-as-a-Judge (LAAJ) rubric and used to steer subsequent tests toward the agent's weakest capabilities. On a travel planner and a Wikipedia writer, the ATA surfaces more diverse and severe failures than expert annotators while matching severity, and finishes in 20--30 minutes versus ten-annotator rounds that took days. Ablating code analysis and web search increases variance and miscalibration, underscoring the value of evidence-grounded test generation. The ATA outputs quantitative metrics and qualitative bug reports for developers. We release the full methodology and open-source implementation for reproducible agent testing: https://github.com/KhalilMrini/Agent-Testing-Agent
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。