评测6大主流代码生成模型，发现GPT-4.1和DeepSeek-R1表现最优。
Holistic Evaluation of State-of-the-Art LLMs for Code Generation
- 基于944道真实LeetCode题，评估多语言代码生成正确性与效率。
- GPT-4.1与DeepSeek-R1在编译、运行、逻辑错误率上均领先。
- 指出提示工程与人工校验对提升生成质量至关重要。
本研究对六种前沿大语言模型（包括通用与代码专用模型）进行了全面的实证评估，涵盖五种编程语言的944道真实世界LeetCode题目。通过编译错误、运行时错误、功能失败及算法次优性等严格指标衡量性能。结果表明，DeepSeek-R1与GPT-4.1在正确性、效率与鲁棒性方面显著优于其他模型。案例分析揭示了语法错误、逻辑缺陷与算法低效等常见失败模式，强调提示工程与人工监督在提升结果中的关键作用。基于此，为开发者提供可操作建议：成功部署需精心选型、优化提示设计，并结合上下文使用，以确保真实软件开发中代码生成的可靠性。
原文摘要 · Abstract (English)
This study presents a comprehensive empirical evaluation of six state-of-the-art large language models (LLMs) for code generation, including both general-purpose and code-specialized models. Using a dataset of 944 real-world LeetCode problems across five programming languages, we assess model performance using rigorous metrics: compile-time errors, runtime errors, functional failures, and algorithmic suboptimalities. The results reveal significant performance variations, with DeepSeek-R1 and GPT-4.1 consistently outperform others in terms of correctness, efficiency, and robustness. Through detailed case studies, we identify common failure scenarios such as syntax errors, logical flaws, and suboptimal algorithms, highlighting the critical role of prompt engineering and human oversight in improving results. Based on these findings, we provide actionable recommendations for developers and practitioners, emphasizing that successful LLM deployment depends on careful model selection, effective prompt design, and context-aware usage to ensure reliable code generation in real-world software development tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。