提出可扩展的多智能体协作评估框架，揭示信息整合缺陷
Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems
- 设计30个任务、3种通信复杂度的无角色基准测试
- 54种配置下1620次实验发现推理整合阶段普遍失败
- 适合研究多智能体系统协同与可扩展性问题的研究者
大语言模型在多智能体系统中通过分散信息来突破上下文限制，但智能体能否真正利用分布式信息进行可靠计算仍是未知。我们提出SILO-BENCH，一个包含30个算法任务、覆盖三种通信复杂度的无角色基准，评估了54种配置共1620次实验。实验揭示了根本性的通信-推理鸿沟：智能体自发形成任务适配的协作拓扑并主动交换信息，却系统性地无法将分布式状态整合为正确答案。失败集中于推理整合阶段，此时智能体通常已获取足够信息但无法有效整合。该协调开销随规模增长，最终完全抵消并行化收益。结果表明，单纯增加智能体数量无法规避上下文限制，SILO-BENCH为追踪真正协作式多智能体系统的发展提供了基础。代码已开源。
原文摘要 · Abstract (English)
Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agents. Yet whether agents can reliably compute with distributed information, rather than merely exchange it, remains an open question. We introduce SILO-BENCH, a role-agnostic benchmark of 30 algorithmic tasks across three communication complexity levels, evaluating 54 configurations over 1,620 experiments. Our experiments expose a fundamental Communication-Reasoning Gap: agents spontaneously form task-appropriate coordination topologies and exchange information actively, yet systematically fail to synthesize distributed state into correct answers. The failure is localized to the reasoning-integration stage where agents often acquire sufficient information but cannot integrate it. This coordination overhead compounds with scale, eventually eliminating parallelization gains entirely. These findings demonstrate that naively scaling agent count cannot circumvent context limitations, and SILO-BENCH provides a foundation for tracking progress toward genuinely collaborative multi-agent systems. The code is available at https://github.com/jwyjohn/acl26-silo-bench .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。