对比12种AI协作方式，发现对抗式设计最有效，跨模型评审次之。
LLM Consortium for Software Design Refinement: A Controlled Experiment on Multi-Agent Collaboration Topologies
- 用三种大模型分角色协作，对抗性指令让设计质量最高
- 交叉模型评审得分第二，各模型独立评估结果一致
- 不同模型对设计优劣判断差异大，需注意评估偏差
我们开展受控实验，评估12种多智能体大模型协作拓扑在软件架构设计中的表现。采用$2\times2\times2$因子设计（权威性×角色×动态），在8个不同复杂度的设计任务上共执行520次实验，每项任务重复5次。设计由三个独立的自动化评估器（GPT-OSS 120B、Claude Opus 4.6、Claude Sonnet 4.6）按12维标准评分。主要发现：第一，结构化对抗式（v4b）在集成评分中排名第一（加权平均4.637/5.0），通过强制重写而非修补提升质量；第二，跨模型评审胜出，即一个模型生成、另一个模型评审，在所有评估器中均位列第二（加权平均4.606）；第三，评估器多样性本身是关键发现——三者均认定v4b最优、v3最差，但对v2b分歧显著（Claude差异1.44，GPT-OSS差异0.45），揭示不同模型家族对设计质量权重认知不同；第四，并行合并机制根本失效，三者均将其置于底部区间（3.65–3.79），因令牌饥饿与拼接效应所致。加权集成（2×Opus + 2×Sonnet + 1×GPT-OSS）在520次运行中提供稳健排名，并经独立交叉验证确认。
原文摘要 · Abstract (English)
We present a controlled experiment evaluating 12 multi-agent LLM collaboration topologies for software architecture design. Using a $2\times2\times2$ factorial design (Authority $\times$ Roles $\times$ Dynamics), we conducted 520 experimental runs across 8 design tasks of varying complexity, with 5 repetitions each. Designs were evaluated on a 12-dimensional rubric by three independent automated evaluators (GPT-OSS 120B, Claude Opus 4.6, Claude Sonnet 4.6). We report four core findings. First, structural adversarial (v4b) ranks #1 by ensemble -- a prompt-engineered adversarial variant that demands rewrite mandates rather than patches (weighted ensemble: 4.637/5.0). Second, cross-model review wins unanimously at #2 -- generate with one model, review with another -- ranking #2 by all three evaluators (weighted ensemble: 4.606). Third, evaluator diversity is itself a finding -- all three evaluators agree v4b is best and v3 is worst, but disagree sharply on v2b (Claude d=1.44 vs. GPT-OSS d=0.45), revealing how different model families weight design qualities. Fourth, parallel merge is fundamentally broken -- all three evaluators place merge variants in the bottom tier (3.65-3.79), due to token starvation and the Frankenstein effect. The weighted ensemble ($2\times$Opus + $2\times$Sonnet + $1\times$GPT-OSS) provides robust rankings across 520 runs, confirmed through independent cross-validation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。