多智能体协作研究的性能差异可能源于实验设计，而非算法优劣。
How Much Coordination Gain Is Real? A Paired Noise-Floor Protocol for Multi-Agent LLM Benchmarks
- 用配对实验对比相同模型在不同协议下的表现，控制输入配置一致
- 发现多数协作优势在重复实验中无法复现，最大差距仅18个百分点
- 提出应只报告机制真正起作用时的表现，避免误导性结论
多智能体大模型协作研究常以微小基准差异证明架构优劣。但一个关键问题：当两个协议在相同模型和基准上配置等价（通过代码审查与SHA-256字节审计确认）时，其初始轮次（trial 0）的差异有多大？在Claude Haiku 4.5与tau^2-bench零售数据集上，无协作（no_coord）与拦截（intercept）两种配置在两个随机种子（n=100）下分别产生+10pp和0pp的符号配对差距，合并后为+5pp（Wilson置信区间[-2,+12]），不显著。最大单种子差距为+18pp（p_corr=0.012），但在第二个种子中变为-3pp（p_corr=1.0），未通过贝叶斯校正。所有种子及合并结果均无显著差异。观察到的配对差距范围为[-3,+18]pp，合并上限威尔逊置信区间约15pp。最近十项多智能体协作架构的宣称效果均低于此局部基线，其中一项也处于该范围之内；这些效果是否能在同模型配对复现，原始设置中未被验证。本文定义协调激活的最小报告协议pass^k（仅包含机制逻辑激活的试次），并设定样本量目标与运行时钩子。实验基于ET-MCP（符合MCP 2026-07-28规范的任务限定负知识存储）进行，用于隔离读者侧选择，非核心贡献。在Haiku 4.5上，候选读者（pull, intercept）未能提升第一轮恢复率，并初步诊断失败原因及现有生产钩子改进方向。
原文摘要 · Abstract (English)
Multi-agent LLM coordination papers report small benchmark deltas as evidence that one architecture beats another. A prior question: how much paired trial-0 disagreement do two protocols produce on the same model and benchmark when their API inputs are configuration-equivalent (matched by code inspection plus a SHA-256 byte audit), short of full identity-replay? On Claude Haiku 4.5 against tau^2-bench retail, the clean configuration-equivalent contrast (no_coord vs. intercept, both inert at trial 0) gives signed paired gaps of +10pp and 0pp across two n=100 seeds; pooled across both, +5pp with Wilson CI [-2,+12], not significant. The largest single-seed contrast (+18pp pull-vs-intercept, p_corr=0.012) did not reproduce at the second seed (-3pp, p_corr=1.0); no trial-0 contrast is significant after Bonferroni at either seed or pooled. The envelope of observed paired gaps spans [-3,+18]pp across two seeds, with pooled upper Wilson CI ~15pp. Seven of ten recent multi-agent coordination architectures report headline effects below this local floor, and one more sits inside the envelope; whether they survive a same-model paired replication is, by construction, untested in their original settings. We define coordination-active pass^k, pass^k restricted to trials where the coordination mechanism is logically active, as the minimum reporting protocol, with sample-size targets and runtime hooks in the body. Measurements run on ET-MCP, a task-scoped negative-knowledge store conformant with MCP 2026-07-28, used as a substrate to isolate reader-side choices, not as a contribution. On Haiku 4.5 the candidate readers (pull, intercept) do not improve trial-1 recovery; we give a preliminary diagnosis of failure modes with refinements on existing production hook surfaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。