剖析大模型代码生成短板，揭示四大失败模式与任务陷阱
Where Do LLMs Still Struggle? An In-Depth Analysis of Code Generation Benchmarks
- 跨四大基准测试识别出大模型高频失败任务
- 发现代码静态复杂度与任务失败存在关联性
- 适合研究模型局限性与评测体系优化的学者
大型语言模型（LLMs）在代码生成方面已取得显著进展，排行榜和基准测试成为评估其性能的核心工具。然而，这些评测难以揭示模型持续失败的任务类型，而此类信息对理解当前瓶颈、推动更强大模型研发至关重要。本文分析了四个主流基准中的代码生成任务，识别出大模型最易失败的场景。通过考察解决方案代码的静态复杂度是否影响表现，并系统审查114个模型反复出错的任务，我们发现了四种重复出现的模型弱点，以及导致失败的常见任务设计缺陷。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable success in code generation, and the race to improve their performance has become a central focus of AI research. Benchmarks and leaderboards are increasingly popular, offering quantitative rankings of LLMs. However, they provide limited insight into the tasks that LLMs consistently fail to solve - information that is crucial for understanding current limitations and guiding the development of more capable models. To address this gap, we examined code generation tasks across four popular benchmarks, identifying those that major LLMs are most likely to fail. To understand the causes of these failures, we investigated whether the static complexity of solution code contributes to them, followed by a systematic inspection of 114 tasks that LLMs consistently struggled with. Our analysis revealed four recurring patterns of weaknesses in LLMs, as well as common complications within benchmark tasks that most often lead to failure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。