提出真实场景下的多智能体编程评估基准,揭示协作模式对开发效率的关键影响。
An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

- 构建真实全栈项目基准MSEval,通过分层需求与确定性评分衡量表现
- 100次实验显示不同协作拓扑导致性能差异超30分,运行时间翻倍
- 结构化流水线最优,过度管理反而降低效率,适合研究多智能体协同
多智能体编程有望加速软件开发,但现有基准依赖合成环境,忽略实际时间和成本,混淆推理与通信,并仅奖励表面完成。本文提出多智能体从零开始评估基准MSEval,基于10个真实全栈项目覆盖10个领域,采用分层需求与确定性评分体系。其执行引擎LegoGent测试10种协作拓扑,智能体通过周期性同步协作,并通过原生CI/CD流水线部署。自动化评分器TAgent动态探测实现,联合测量功能正确性、延迟与前缀缓存词元成本。在100次运行中,发现组织拓扑在速度-成本-质量权衡中与模型能力相当:相同任务与模型下,拓扑变化使得分波动超过30分,运行时间翻倍。结构化流水线收敛最快且质量最高,而过度管理显著降低性能。MSEval建立了可复现的多智能体实际软件构建评估标准,代码已开源。
原文摘要 · Abstract (English)
Multi-agent vibe coding promises to accelerate software development, yet existing benchmarks rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. We introduce multi-agent from-scratch evaluation benchmark, MSEval, evaluating multi-agent coding on real-world tasks. Grounded in 10 authentic, full-stack projects across 10 domains, MSEval scores performance using hierarchical requirements and deterministic rubrics. Its execution engine, LegoGent, tests 10 collaboration topologies where agents coordinate via periodic sync intervals and deploy through native CI/CD pipelines. Concurrently, the automated grader TAgent dynamically probes implementations to jointly measure functional success, latency, and prefix-cached token cost. Across 100 runs, MSEval reveals that organizational topology rivals model capability in shaping the speed--cost--quality trade-off. For identical tasks and models, varying the topology shifts scores by over 30 points and doubles wall-clock time. Structured pipelines converge fastest with the highest quality, whereas heavy managerial oversight degrades performance. Ultimately, MSEval establishes a rigorous, reproducible standard for measuring how multi-agent teams actually build software. The benchmark is released at https://github.com/robinren03/MSEval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。