让大模型智能体测试可自动化,提升质量与效率
Automated structural testing of LLM-based agents: methods, framework, and case studies
- 用追踪与模拟技术捕获智能体行为轨迹,实现可复现测试
- 支持自动验证、快速定位问题,测试覆盖率更高
- 适合研发团队在开发流程中集成自动化测试
基于大语言模型的智能体正被广泛应用于多个领域。由于其在无监督环境下与用户交互,必须进行充分测试。现有测试方法主要从用户视角开展验收级评估,虽直观但依赖人工、难以自动化、不利于根因分析且测试环境成本高。本文提出结构化测试方法,利用 OpenTelemetry 追踪记录智能体行为轨迹,通过模拟技术确保大模型输出可复现,并加入断言实现测试自动验证。该方法使智能体组件与交互可在自动化流程中进行深度技术级测试。案例研究表明,结构化测试可有效引入软件工程最佳实践,如测试金字塔、回归测试、测试驱动开发和多语言测试。实测中实现测试自动执行与更快的根因分析,显著降低测试成本,提升智能体质量。项目已开源,提供 GitHub 参考实现。
原文摘要 · Abstract (English)
LLM-based agents are rapidly being adopted across diverse domains. Since they interact with users without supervision, they must be tested extensively. Current testing approaches focus on acceptance-level evaluation from the user's perspective. While intuitive, these tests require manual evaluation, are difficult to automate, do not facilitate root cause analysis, and incur expensive test environments. In this paper, we present methods to enable structural testing of LLM-based agents. Our approach utilizes traces (based on OpenTelemetry) to capture agent trajectories, employs mocking to enforce reproducible LLM behavior, and adds assertions to automate test verification. This enables testing agent components and interactions at a deeper technical level within automated workflows. We demonstrate how structural testing enables the adaptation of software engineering best practices to agents, including the test automation pyramid, regression testing, test-driven development, and multi-language testing. In representative case studies, we demonstrate automated execution and faster root-cause analysis. Collectively, these methods reduce testing costs and improve agent quality through higher coverage, reusability, and earlier defect detection. We provide an open source reference implementation on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。