对比10个代码大模型，发现组合使用可提升性能83%。
Wisdom and Delusion of LLM Ensembles for Code Generation and Repair
- 用多种模型组合生成代码，比单模型更优
- 共识策略易选错，多样性策略能实现95%理论上限
- 小规模组合也能显著提效，适合资源有限团队
当前追求单一大型语言模型完成所有软件工程任务，成本高昂且忽视了不同模型间的互补潜力。然而，代码类大模型的互补程度及最优集成策略尚不明确，导致实践者难以超越单模型系统。为此，我们对来自五个家族的十个独立大模型，以及三种集成方式，在三个涵盖代码生成与程序修复的基准上进行了实证比较。评估了模型间的互补性，以及最佳单模型与集成系统之间的性能差距。进一步测试了多种选择策略以从集成候选结果中选出正确解。研究发现，集成系统的理论上限性能可比最佳单模型高出83%。结果表明，基于共识的策略会陷入‘流行陷阱’，放大常见但错误的输出；而基于多样性的策略则能达到理论上限的95%，且在仅两个模型的小型集成中也有效，为低成本提升性能提供了可行路径。
原文摘要 · Abstract (English)
Today's pursuit of a single Large Language Model (LMM) for all software engineering tasks is resource-intensive and overlooks the potential benefits of complementarity, where different models contribute unique strengths. However, the degree to which coding LLMs complement each other and the best strategy for maximizing an ensemble's potential are unclear, leaving practitioners without a clear path to move beyond single-model systems. To address this gap, we empirically compare ten individual LLMs from five families, and three ensembles of these LLMs across three software engineering benchmarks covering code generation and program repair. We assess the complementarity between models and the performance gap between the best individual model and the ensembles. Next, we evaluate various selection heuristics to identify correct solutions from an ensemble's candidate pool. We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model. Our results show that consensus-based strategies for selecting solutions fall into a "popularity trap," amplifying common but incorrect outputs. In contrast, a diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles, enabling a cost-efficient way to enhance performance by leveraging multiple LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。