用主动干预验证错误假设,让大模型多智能体系统失败后自动修复。
DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
- 通过编辑消息、改计划等干预手段主动验证故障假说。
- 在多个数据集上将18%-28%的失败任务转为成功,推进16%里程碑进度。
- 适合关注大模型系统可靠性与可调试性的研究者和工程师。
基于大语言模型(LLM)的多智能体系统难以调试,因故障常源于长而分支的交互轨迹。现有方法依赖日志定位错误,仅归因于单一智能体或步骤,存在两大缺陷:(i) 仅靠日志生成假说缺乏验证,(ii) 单一归因常不成立,我们发现多个不同干预可独立修复任务。为此,提出DoVer框架,通过针对性干预(如编辑消息、修改计划)主动验证假说,并以任务是否成功或进展作为评判标准,转向结果导向的调试。在Magnetic-One框架下,基于GAIA和AssistantBench数据集,DoVer使18%-28%失败试验转为成功,实现最高16%里程碑进度提升,验证或否决30%-60%故障假说。在GSMPlus数据集和AG2框架中,亦恢复49%失败试验。结果表明,干预是提升智能体系统可靠性的有效机制,为更鲁棒、可扩展的调试方法开辟路径。
原文摘要 · Abstract (English)
Large language model (LLM)-based multi-agent systems are challenging to debug because failures often arise from long, branching interaction traces. The prevailing practice is to leverage LLMs for log-based failure localization, attributing errors to a specific agent and step. However, this paradigm has two key limitations: (i) log-only debugging lacks validation, producing untested hypotheses, and (ii) single-step or single-agent attribution is often ill-posed, as we find that multiple distinct interventions can independently repair the failed task. To address the first limitation, we introduce DoVer, an intervention-driven debugging framework, which augments hypothesis generation with active verification through targeted interventions (e.g., editing messages, altering plans). For the second limitation, rather than evaluating on attribution accuracy, we focus on measuring whether the system resolves the failure or makes quantifiable progress toward task success, reflecting a more outcome-oriented view of debugging. Within the Magnetic-One agent framework, on the datasets derived from GAIA and AssistantBench, DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses. DoVer also performs effectively on a different dataset (GSMPlus) and agent framework (AG2), where it recovers 49% of failed trials. These results highlight intervention as a practical mechanism for improving reliability in agentic systems and open opportunities for more robust, scalable debugging methods for LLM-based multi-agent systems. Project website and code will be available at https://aka.ms/DoVer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。