arXiv:2601.12951cs.SEcs.AI2026-01

大模型理解代码的能力与人类指标无关,展现独特规律。

Beyond Accuracy: Characterizing Code Comprehension Capabilities in (Large) Language Models

  • 将代码理解转为输入输出一致性判断,实现模型诊断
  • 人类复杂度指标与模型表现相关性低(AUROC 0.63)
  • 模型自身规律可被学习模型捕捉(AUROC 0.86),适合开发者参考

大型语言模型(LLMs)正日益融入软件工程流程,但现有基准仅提供粗略性能汇总,掩盖了模型能力与局限的多样性。本文探究大模型的代码理解能力是否符合传统以人为中心的软件度量标准,或反映非人类特有的规律。我们提出一种诊断框架,将代码理解重构为二元输入-输出一致性任务,适用于分类与生成模型评估。基于大规模数据集,我们将模型表现与传统人类中心复杂度指标(如词法规模、控制流复杂度、抽象语法树结构)进行关联分析。结果显示,人类定义的指标与大模型成功度的相关性极低(AUROC 0.63),而影子模型则显著提升预测性能(AUROC 0.86),揭示出超越传统软件度量的复杂、部分可预测模式。这表明大模型的代码理解体现的是仅部分可通过人为设计或学习特征捕捉的模型特有规律,强调需发展超越整体准确率的实例级诊断方法,并承认对正确结果预测的根本性限制。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly integrated into software engineering workflows, yet current benchmarks provide only coarse performance summaries that obscure the diverse capabilities and limitations of these models. This paper investigates whether LLMs' code-comprehension performance aligns with traditional human-centric software metrics or instead reflects distinct, non-human regularities. We introduce a diagnostic framework that reframes code understanding as a binary input-output consistency task, enabling the evaluation of classification and generative models. Using a large-scale dataset, we correlate model performance with traditional, human-centric complexity metrics, such as lexical size, control-flow complexity, and abstract syntax tree structure. Our analyses reveal minimal correlation between human-defined metrics and LLM success (AUROC 0.63), while shadow models achieve substantially higher predictive performance (AUROC 0.86), capturing complex, partially predictable patterns beyond traditional software measures. These findings suggest that LLM comprehension reflects model-specific regularities only partially accessible through either human-designed or learned features, emphasizing the need for benchmark methodologies that move beyond aggregate accuracy and toward instance-level diagnostics, while acknowledging fundamental limits in predicting correct outcomes.

代码理解大模型评估软件度量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。