用可配置多智能体框架自动测试大模型智能体，效率提升10倍以上。
Configurable multi-agent framework for scalable and realistic testing of llm-based agents
- 构建问答生成与评估双智能体，通过共享上下文模块化控制测试场景。
- 在真实聊天机器人测试中发现3.3%的边缘错误，接近专家水平的5.8%。
- 支持动态对话演化，适合高并发、高保真度的大模型系统验证。
大语言模型（LLM）智能体行为复杂且依赖上下文，使静态基准和人工测试迅速过时。我们提出Neo——一个可配置的多智能体框架，实现对基于LLM系统的自动化、多轮、高仿真评估。Neo通过共享上下文枢纽连接问答生成智能体与评估智能体，支持领域提示、场景控制和动态反馈的模块化组合。测试输入从涵盖对话流程、用户意图和情绪基调的概率状态模型中采样，实现多样化、类人对话，并随每轮互动自适应调整。应用于生产级卖家财务助手聊天机器人，Neo（i）在五类攻击下识别出3.3%的边缘故障，接近专家红队人员5.8%的发现率；（ii）吞吐量提升10-12倍，45分钟生成180个连贯测试问题，相当于人工16小时工作量。此外，其随机策略平衡了话题覆盖与对话深度，比手动脚本探索更广泛的行为空间。因此，Neo为可扩展、自我演化的LLM质量保证奠定了基础：其智能体接口、状态控制器和反馈环路具备模型无关性，可扩展至更丰富的事实对齐与合规检查。我们已开源该框架，以促进新兴代理系统的可复现、高保真测试。
原文摘要 · Abstract (English)
Large-language-model (LLM) agents exhibit complex, context-sensitive behaviour that quickly renders static benchmarks and ad-hoc manual testing obsolete. We present Neo, a configurable, multi-agent framework that automates realistic, multi-turn evaluation of LLM-based systems. Neo couples a Question Generation Agent and an Evaluation Agent through a shared context-hub, allowing domain prompts, scenario controls and dynamic feedback to be composed modularly. Test inputs are sampled from a probabilistic state model spanning dialogue flow, user intent and emotional tone, enabling diverse, human-like conversations that adapt after every turn. Applied to a production-grade Seller Financial Assistant chatbot, Neo (i) uncovered edge-case failures across five attack categories with a 3.3% break rate close to the 5.8% achieved by expert human red-teamers, and (ii) delivered 10-12X higher throughput, generating 180 coherent test questions in around 45 mins versus 16h of human effort. Beyond security probing, Neo's stochastic policies balanced topic coverage and conversational depth, yielding broader behavioural exploration than manually crafted scripts. Neo therefore lays a foundation for scalable, self-evolving LLM QA: its agent interfaces, state controller and feedback loops are model-agnostic and extensible to richer factual-grounding and policy-compliance checks. We release the framework to facilitate reproducible, high-fidelity testing of emerging agentic systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。