提出新方法诊断大模型多智能体系统失败,发现传统重跑效果差。
Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
- 构建可控实验框架SymTrace,精准复现失败场景
- 实测现有重跑方法修复率仅6.90%,失败重现率67.97%
- 症状驱动干预提升修复率至20.15%,适合系统调试研究者
随着基于大语言模型(LLM)的多智能体系统(MASs)在长时程复杂任务中应用日益广泛,其可靠性已成为阻碍实际部署的核心瓶颈。现有MAS调试与修复方法通常依赖于重跑并重新采样整个执行轨迹。然而一个根本问题仍未解决:这些方法是否真正因果性地修复了失败,还是仅通过LLM采样的随机性偶然修复?为评估修复方法的有效性,我们引入SymTrace——一个受控评估框架,可记录MAS执行轨迹并建立干预锚点。在重播时,它能利用记录日志重建锚点前的执行过程,仅再生下游轨迹,从而实现故障的可靠重现。我们进一步构建了数据集SymFail,包含536条人工标注的失败轨迹,附带图结构位置、类别和追踪证据。基于此,我们在三个主流MAS框架上开展大规模实证研究。结果表明,现有无引导重跑方法极不可靠,失败重现率与修复率分别仅为67.97%和6.90%。在此基础上,我们探索症状驱动干预方法,成功修复20.15%的失败案例(相较最先进方法提升191.89%)。本研究旨在为MAS调试与修复研究提供可操作洞见,推动多智能体系统的稳健部署。
原文摘要 · Abstract (English)
As large language model (LLM)-based multi-agent systems (MASs) are increasingly applied to long-horizon complex tasks, their reliability has emerged as the core bottleneck hindering their real-world deployment. Existing MAS debugging and repair methods typically rely on rerunning and resampling the entire execution trajectory. However, a fundamental question remains to be answered: do these methods causally repair MAS failures or merely stochastically repair by leveraging the randomness of LLM sampling? To evaluate the effectiveness of MAS repair methods, we introduce SymTrace, a controlled evaluation framework that records the MAS execution trajectory and establishes intervention anchors. During replay, it effectively reconstructs the execution before the anchor using recorded logs and only regenerates the downstream trajectory, thereby enabling the reliable reproduction of MAS failures. We further construct the dataset SymFail, comprising 536 human-annotated failure trajectories with graph-linked locations, categories, and trace evidence. Based on these foundations, we conduct a large-scale empirical study across three mainstream MAS frameworks. Our findings reveal that existing unguided rerun methods are highly unreliable, exhibiting low failure reproduction and repair rates (only 67.97% and 6.90%, respectively). Building upon these findings, we further explore the effectiveness of a symptom-driven intervention method, which successfully repairs 20.15% of the failed cases (a 191.89% improvement to state-of-the-art repair methods). This study aims to provide actionable insights for MAS debugging and repair research, paving the way for the robust deployment of multi-agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。