用高效方法检测AI代理流程是否退化，成本降七成以上。
AgentAssay: Token-Efficient Regression Testing for Non-Deterministic AI Agent Workflows
- 通过统计检验生成三种判定结果，兼顾准确与效率
- 行为指纹技术使退化检测率从0提升至86%
- 适合部署在持续集成中的大型AI系统验证
自主AI代理已大规模部署，但尚无系统方法验证其在提示、工具、模型或编排逻辑变更后的稳定性。我们提出AgentAssay，首个面向非确定性AI代理工作流的高性价比回归测试框架，在保持严格统计保障的前提下实现78%-100%的成本降低。核心贡献包括：(1) 基于假设检验的三值判定（通过/失败/不确定）；(2) 五维代理覆盖度量；(3) 代理特异性突变测试算子；(4) 代理工作流的元变换关系；(5) 将CI/CD部署门禁作为统计决策程序；(6) 行为指纹技术将执行轨迹映射为紧凑向量，实现多变量退化检测；(7) 自适应预算优化，根据行为方差动态调整试验次数；(8) 基于追踪的离线分析，实现生产数据零成本测试。在5个模型（GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4）、3种场景和7,605次试验中，行为指纹实现86%的检测能力，而二值测试为0%；SPRT将试验次数减少78%；完整流水线通过追踪先行分析实现100%成本节省。实现：20,000+行Python代码，751个测试，10个框架适配器。
原文摘要 · Abstract (English)
Autonomous AI agents are deployed at unprecedented scale, yet no principled methodology exists for verifying that an agent has not regressed after changes to its prompts, tools, models, or orchestration logic. We present AgentAssay, the first token-efficient framework for regression testing non-deterministic AI agent workflows, achieving 78-100% cost reduction while maintaining rigorous statistical guarantees. Our contributions include: (1) stochastic three-valued verdicts (PASS/FAIL/INCONCLUSIVE) grounded in hypothesis testing; (2) five-dimensional agent coverage metrics; (3) agent-specific mutation testing operators; (4) metamorphic relations for agent workflows; (5) CI/CD deployment gates as statistical decision procedures; (6) behavioral fingerprinting that maps execution traces to compact vectors, enabling multivariate regression detection; (7) adaptive budget optimization calibrating trial counts to behavioral variance; and (8) trace-first offline analysis enabling zero-cost testing on production traces. Experiments across 5 models (GPT-5.2, Claude Sonnet 4.6, Mistral-Large-3, Llama-4-Maverick, Phi-4), 3 scenarios, and 7,605 trials demonstrate that behavioral fingerprinting achieves 86% detection power where binary testing has 0%, SPRT reduces trials by 78%, and the full pipeline achieves 100% cost savings through trace-first analysis. Implementation: 20,000+ lines of Python, 751 tests, 10 framework adapters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。