发现代码生成多智能体系统存在严重鲁棒性缺陷,提出修复方案提升稳定性。
Understanding and Bridging the Planner-Coder Gap: A Systematic Study on the Robustness of Multi-Agent Systems for Code Generation
- 通过语义保持变异测试,揭示多阶段转换中的信息丢失问题。
- 同一逻辑输入下,系统成功率下降7.9%至83.3%,主因是规划-编码断层。
- 提出多提示生成与监控代理机制,可修复40.0%至88.9%的失败案例。
多智能体系统(MASs)在自动化代码生成任务中表现优异,但其鲁棒性的内在机制仍不清晰,限制了实际应用。本文采用基于变异的系统性实证研究方法,设计包含语义保持变异算子和新适应度函数的测试流程,评估主流MASs在多个数据集和LLMs上的表现。结果发现显著鲁棒性缺陷:语义等价输入导致性能急剧下降,原能解决的问题中7.9%–83.3%无法完成。深入分析表明,根本原因在于“规划-编码断层”,占所有失败的75.3%。该断层源于规划阶段将需求分解为不充分计划,编码阶段由此产生逻辑误解。基于此信息转换模型,本文提出修复方法,通过多提示生成减少信息损失,并引入监控代理以弥合断层。实验验证,该方法有效提升鲁棒性,修复40.0%–88.9%的已识别失败。本工作揭示了MASs的关键脆弱性并提供有效缓解策略,为构建更可靠的代码生成系统提供关键洞见。
原文摘要 · Abstract (English)
Multi-agent systems (MASs) have emerged as a promising paradigm for automated code generation, demonstrating impressive performance on established benchmarks. Despite their prosperous development, the fundamental mechanisms underlying their robustness remain poorly understood, raising critical concerns for real-world deployment. This paper conducts a systematic empirical study to uncover the internal robustness flaws of MASs using a mutation-based methodology. By designing a testing pipeline incorporating semantic-preserving mutation operators and a novel fitness function, we assess mainstream MASs across multiple datasets and LLMs. Our findings reveal substantial robustness flaws: semantically equivalent inputs cause drastic performance drops, with MASs failing to solve 7.9\%--83.3\% of problems they initially resolved successfully. Through comprehensive failure analysis, we discover a fundamental cause underlying these robustness issues: the \textit{planner-coder gap}, which accounts for 75.3\% of failures. This gap arises from information loss in the multi-stage transformation process where planning agents decompose requirements into underspecified plans, and coding agents subsequently misinterpret intricate logic during code generation. Based on this formulated information transformation process, we propose a \textit{repairing method} that mitigates information loss through multi-prompt generation and introduces a monitor agent to bridge the planner-coder gap. Evaluation shows that our repairing method effectively enhances the robustness of MASs by solving 40.0\%--88.9\% of identified failures. Our work uncovers critical robustness flaws in MASs and provides effective mitigation strategies, contributing essential insights for developing more reliable MASs for code generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。