用三维度评估大模型编程能力，发现其代码思维与人类有本质差异。
Coding Triangle: How Does Large Language Model Understand Code?
- 构建编辑分析、代码实现、测试生成三维度评估框架
- 模型解法缺乏多样性，错误集中且受训练数据偏见影响
- 融合人类解题思路可显著提升模型性能与鲁棒性
大型语言模型（LLMs）在代码生成方面取得显著进展，但其真正的编程能力仍不明确。我们提出代码三角框架，从编辑分析、代码实现和测试用例生成三个基本维度系统评估LLMs。在编程竞赛基准上的大量实验表明，尽管模型在三者间能形成自洽体系，其解法却往往缺乏人类程序员的多样性和鲁棒性。我们发现模型认知与人类专长之间存在显著分布偏差，模型错误常因训练数据偏差而聚集，且推理迁移能力有限。研究显示，引入人类生成的题解、代码及多样化测试用例，并采用模型混合，可显著提升LLM的性能与鲁棒性。此外，我们揭示了模型认知中的自洽与不一致特性，或可为自我反思与自我改进提供方向，助力更强大的编码模型发展。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved remarkable progress in code generation, yet their true programming competence remains underexplored. We introduce the Code Triangle framework, which systematically evaluates LLMs across three fundamental dimensions: editorial analysis, code implementation, and test case generation. Through extensive experiments on competitive programming benchmarks, we reveal that while LLMs can form a self-consistent system across these dimensions, their solutions often lack the diversity and robustness of human programmers. We identify a significant distribution shift between model cognition and human expertise, with model errors tending to cluster due to training data biases and limited reasoning transfer. Our study demonstrates that incorporating human-generated editorials, solutions, and diverse test cases, as well as leveraging model mixtures, can substantially enhance both the performance and robustness of LLMs. Furthermore, we reveal both the consistency and inconsistency in the cognition of LLMs that may facilitate self-reflection and self-improvement, providing a potential direction for developing more powerful coding models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。