评测11个大模型在5类编程任务中的表现，发现效率差异巨大。
Comprehensive Evaluation of Large Language Models on Software Engineering Tasks: A Multi-Task Benchmark
- 构建多任务自动化评估框架，覆盖编码、修复、重构等五类任务。
- 完美完成任务的模型间耗时差22倍，工具使用效率差49倍。
- 揭示两种低效模式，研究类任务成功率仅90.9%需关注。
大语言模型在软件工程领域展现出强大能力，但涵盖多样开发活动的全面评估基准仍有限。本文对11个前沿大模型在五类典型软件工程任务（缺陷修复、功能开发、代码重构、技术写作、研究综述）上进行多任务评估。基于自动化验证框架，衡量输出质量与完成效率。关键发现：(1) 完全正确完成任务的模型间，完成时间差异达22倍，工具使用效率相差49倍，预估成本相差53倍；(2) 工具调用频率与成功无相关性（r = 0.077, p = 0.575），同一任务中模型调用工具次数从3次到917次不等；(3) 识别出两类低效模式：循环冗余与推理低效；(4) 编码任务成功率100%，研究类任务挑战更大（成功率90.9%）。所有实验数据、验证脚本与分析代码均已公开，确保可复现性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities in software engineering, yet comprehensive benchmarks covering diverse SE activities remain limited. We present a multi-task evaluation of 11 state-of-the-art LLMs across five representative software engineering tasks: bug fixing, feature development, code refactoring, technical copywriting, and research synthesis. Our automated verification framework measures both output quality and completion efficiency. Key findings reveal that (1) models achieving identical perfect scores exhibit 22x variation in completion time, 49x variation in tool efficiency, and 53x variation in estimated cost; (2) tool usage frequency shows no correlation with success (r = 0.077, p = 0.575) - one model used 917 tool calls while another solved the same task with 3 calls; (3) we identify two distinct inefficiency patterns: loop inefficiency and inference inefficiency; and (4) coding tasks achieve 100 percent success while research tasks present greater challenges (90.9 percent). We release all experimental data, verification scripts, and analysis code for full reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。