自动生成多智能体系统错误数据,解决调试难问题。
Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
- 用大模型动态注入上下文相关的故障到正常轨迹中
- 构建9533条带标签轨迹,覆盖多种架构与任务
- 支持三种学习范式,小模型性能媲美大厂超大规模模型
基于大语言模型的多智能体系统在解决复杂问题上取得显著进展,但其能力提升也带来了结构性脆弱,难以调试。当前提升可靠性的主要障碍是缺乏大规模、多样化的错误归因数据集,现有资源依赖昂贵且不可扩展的人工标注。为此,我们提出Aegis框架,实现多智能体系统的自动化错误生成与归因。Aegis通过基于大模型的操控器,自适应地在成功执行轨迹中注入上下文感知的错误,构建包含9533条轨迹的大型数据集,每条均标注故障智能体和错误类型,覆盖多种多智能体架构与任务领域。利用细粒度标签及正负样本对的结构化设计,Aegis支持监督微调、强化学习和对比学习三种学习范式,并为每种范式开发了相应方法。全面实验表明,训练模型在错误归因任务中均有显著提升。值得注意的是,部分微调后的语言模型性能达到甚至超过规模大一个数量级的专有模型,验证了该自动化数据生成框架在提升多智能体系统鲁棒性与可解释性方面的关键价值。
原文摘要 · Abstract (English)
Large language model based multi-agent systems (MAS) have unlocked significant advancements in tackling complex problems, but their increasing capability introduces a structural fragility that makes them difficult to debug. A key obstacle to improving their reliability is the severe scarcity of large-scale, diverse datasets for error attribution, as existing resources rely on costly and unscalable manual annotation. To address this bottleneck, we introduce Aegis, a novel framework for Automated error generation and attribution for multi-agent systems. Aegis constructs a large dataset of 9,533 trajectories with annotated faulty agents and error modes, covering diverse MAS architectures and task domains. This is achieved using a LLM-based manipulator that can adaptively inject context-aware errors into successful execution trajectories. Leveraging fine-grained labels and the structured arrangement of positive-negative sample pairs, Aegis supports three different learning paradigms: Supervised Fine-Tuning, Reinforcement Learning, and Contrastive Learning. We develop learning methods for each paradigm. Comprehensive experiments show that trained models consistently achieve substantial improvements in error attribution. Notably, several of our fine-tuned LLMs demonstrate performance competitive with or superior to proprietary models an order of magnitude larger, validating our automated data generation framework as a crucial resource for developing more robust and interpretable multi-agent systems. Our project website is available at https://kfq20.github.io/Aegis-Website/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。