提出评测框架，帮大模型多智能体系统选对通信协议
ProtocolBench: Which LLM MultiAgent Protocol to Choose?
- 构建四维评测体系，量化比较不同协议的性能表现
- 实测显示协议差异可导致完成时间差36.5%、延迟差3.48秒
- 开发可学习的协议路由器，按场景自动选最优协议
随着大规模多智能体系统的发展，通信协议层已成为影响性能与可靠性的关键因素，但现有协议（A2A、ACP、ANP、Agora等）的选择多依赖直觉，缺乏标准化评估。我们提出ProtocolBench，一个系统性评估代理协议的基准，涵盖任务成功率、端到端延迟、消息或字节开销以及故障下的鲁棒性四个维度。在Streaming Queue场景中，不同协议的总完成时间相差达36.5%，平均端到端延迟差异为3.48秒；在Fail-Storm Recovery测试中，各协议的恢复能力也显著不同。此外，我们设计了ProtocolRouter，一种基于需求和运行时信号的可学习协议路由机制，相比最佳单协议基线，可将故障恢复时间减少最多18.1%，并在GAIA任务中实现更高成功率。我们还发布了ProtocolRouterBench，以标准化协议评估，提升大规模系统的可靠性。
原文摘要 · Abstract (English)
As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。