arXiv:2605.08647cs.CLcs.AI2026-05被引 1

诊断大模型协作中的隐蔽失效,揭示架构缺陷比模型能力更影响系统可靠性

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

论文配图:AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators
图 1 · 摘自论文原文
  • 构建900个人工验证任务,隔离四类协作风险:指令衰减、错误传播、信息泄露、追踪失效
  • 发现不同模型在各类风险上表现差异显著,且拓扑结构解释了7%-40%的多跳信息丢失
  • 揭示汇聚型图结构存在约束丢弃瓶颈,线性链路无此问题,结构设计决定可靠性

多智能体系统通过同伴协作实现顶尖性能。然而,当流程中某个智能体悄然忽略约束时,最终输出看似正确,但推理链已暗中被破坏,现有基于结果的评估无法察觉此类多跳过程故障。为在部署前量化这些漏洞,我们提出AgentCollabBench,一个包含900个经人工验证的任务基准,覆盖软件工程、DevOps和数据工程领域。每个任务独立测试四类行为风险:指令衰减(约束能否抵御同伴压力?)、假信念传染(错误是否通过共识扩散?)、上下文泄露(任务间信息是否溢出?)、追踪持久性(标记数据能否到达最终代理?)。评估四个现代大模型(GPT 4.1 mini、Gemini 2.5 Flash Lite、Qwen-3.5-35B-A3B、Llama 3.1 8B Instruct)后,暴露了仅凭结果评估无法发现的模型特异性脆弱性:例如,Qwen-3.5-35B-A3B在追踪持久性和指令稳定性上领先,而GPT 4.1 mini在泄露控制和假信念抵抗上表现更好。除模型差异外,通信拓扑成为主要风险因子,解释了7%-40%的多跳信息存活率方差。该效应源于汇聚型有向无环图(converging-DAG)节点特有的合成瓶颈:一个接收多个父节点输入的代理会丢弃少数分支携带的约束,这一结构性缺陷在线性链路中不存在。AgentCollabBench表明,次优拓扑可能悄然抹去高性能模型的防护机制,论证多智能体可靠性根本上是结构问题,单纯提升模型智能无法替代架构优化。

原文摘要 · Abstract (English)

Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint, the system's final output may look correct even though the reasoning chain was quietly corrupted, and existing outcome-based evaluations are blind to such multi-hop process failures. To make these vulnerabilities measurable before deployment, we introduce AgentCollabBench, a diagnostic benchmark of 900 human-validated tasks spanning software engineering, DevOps, and data engineering. Each task isolates one of four behavioral risks: instruction decay (does a constraint survive peer pressure?), false-belief contagion (does a falsehood spread through consensus?), context leakage (does information bleed between tasks?), and tracer durability (does marked data reach the final agent?). Evaluating four modern LLMs (GPT 4.1 mini, Gemini 2.5 Flash Lite, Qwen-3.5-35B-A3B, and Llama 3.1 8B Instruct), we expose model-specific vulnerability profiles invisible to outcome-only evaluation; Qwen-3.5-35B-A3B, for example, leads on tracer durability and instruction stability, while GPT 4.1 mini leads on leakage containment and false-belief resistance. Beyond per-model differences, communication topology emerges as a primary risk factor that explains 7-40% of the variance in multi-hop information survival. The effect traces to a synthesis bottleneck specific to converging-DAG nodes: an agent weighing competing parent inputs discards constraints carried by a minority branch, a bottleneck structurally absent from linear chains. AgentCollabBench demonstrates that suboptimal topology can silently erase the safeguards of highly capable models, arguing that multi-agent reliability is fundamentally a structural problem and that scaling model intelligence alone is no substitute for architecture.

多智能体协作诊断系统可靠性架构设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。