自动定位大模型多智能体系统任务失败的元凶与时机
Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems
- 构建细粒度失败日志数据集，标注失败归属的智能体与关键步骤
- 三种自动化方法中最高识别准确率达53.5%，但定位错误步骤仅14.2%
- 现有顶尖推理模型仍难实用，凸显该任务复杂性
大模型多智能体系统的故障归因——即识别导致任务失败的具体智能体及关键步骤——对系统调试至关重要，但目前研究不足且依赖人工。本文首次提出并定义了该研究方向：大模型多智能体系统的自动化故障归因。为此，我们构建了Who&When数据集，包含127个大模型多智能体系统的详尽失败日志，并对失败事件进行细粒度标注，明确关联到具体智能体和决定性错误步骤。基于该数据集，我们开发并评估了三种自动化故障归因方法，总结其优劣。最佳方法在识别失败责任智能体上达到53.5%准确率，但在精确定位失败步骤上仅14.2%，部分方法甚至低于随机水平。即便采用当前最先进的推理模型（如OpenAI o1和DeepSeek R1），也未能达到实际可用程度。这些结果揭示了该任务的高度复杂性，亟需进一步研究。代码与数据集已开源于https://github.com/mingyin1/Agents_Failure_Attribution。
原文摘要 · Abstract (English)
Failure attribution in LLM multi-agent systems-identifying the agent and step responsible for task failures-provides crucial clues for systems debugging but remains underexplored and labor-intensive. In this paper, we propose and formulate a new research area: automated failure attribution for LLM multi-agent systems. To support this initiative, we introduce the Who&When dataset, comprising extensive failure logs from 127 LLM multi-agent systems with fine-grained annotations linking failures to specific agents and decisive error steps. Using the Who&When, we develop and evaluate three automated failure attribution methods, summarizing their corresponding pros and cons. The best method achieves 53.5% accuracy in identifying failure-responsible agents but only 14.2% in pinpointing failure steps, with some methods performing below random. Even SOTA reasoning models, such as OpenAI o1 and DeepSeek R1, fail to achieve practical usability. These results highlight the task's complexity and the need for further research in this area. Code and dataset are available at https://github.com/mingyin1/Agents_Failure_Attribution
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。