通过感知不确定度提升多智能体协作编程的可靠性
UA-ChatDev: Uncertainty-Aware Multi-Agent Collaboration for Reliable Software Development

- 在多智能体协作中引入令牌级置信度评估,动态识别不可靠输出
- 当不确定性超阈值时自动触发检索验证,减少错误传播
- 在SRDD基准上全面优于现有方法,尤其提升代码可执行性
软件开发是需要多角色协作的复杂任务。大语言模型(LLMs)已推动自主多智能体软件开发框架的发展,通过角色分工实现需求分析、编码、测试与优化的自动化。然而,现有方法通常假设各阶段输出具有同等可靠性,导致幻觉传播问题:早期生成的错误决策会传递至下游,影响最终软件质量。为此,我们提出UA-ChatDev,一种基于不确定度感知的多智能体协作框架,将不确定性量化融入智能体交互。该框架采用轻量级的基于令牌级别对数概率的不确定性估计机制,评估智能体响应的置信度,并结合阶段感知的阈值校准策略,在不确定性超过可接受水平时,选择性触发基于检索的验证。在SRDD基准上的大量实验表明,UA-ChatDev在完整性、可执行性、一致性及整体质量指标上持续优于现有单智能体和多智能体框架。进一步的消融研究与通信分析验证了不确定度感知交互显著提升了代码执行可靠性。
原文摘要 · Abstract (English)
Software development is a complex task that demands cooperation among agents with diverse roles. Large language models (LLMs) have enabled autonomous multi-agent software development frameworks that leverage role-based collaboration to automate requirements analysis, coding, testing, and refinement. However, existing approaches typically assume that intermediate agent outputs are equally reliable, leaving them vulnerable to hallucination propagation, where incorrect decisions generated in early development phases are transferred to downstream agents and negatively impact final software quality. To address this challenge, we propose UA-ChatDev, an uncertainty-aware multi-agent software development framework that integrates uncertainty quantification into agent interactions. It introduces a lightweight uncertainty estimation mechanism based on token-level log probabilities to assess the confidence of agent responses and employs phase-aware threshold calibration to selectively trigger retrieval-based verification when uncertainty exceeds acceptable levels. Extensive experiments on the SRDD benchmark demonstrate that UA-ChatDev consistently outperforms existing single-agent and multi-agent software development frameworks across completeness, executability, consistency, and overall quality metrics. Further ablation studies and communication analyses verify that uncertainty-aware interactions enhance code execution reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。