构建代码静态分析基准,评估大模型深层语义理解能力。
CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks
- 设计语义感知采样策略,覆盖多语言程序的依赖关系
- 测试10个主流大模型,发现其在多步推理任务中表现不佳
- 适合研究代码理解与模型推理机制的开发者参考
大型语言模型已广泛应用于代码生成、程序修复和漏洞检测等软件工程领域,这些应用需要超越表面代码模式的理解,如值传播、控制流和程序元素间的依赖关系。然而,现有基准主要评估端到端结果(如代码是否修复正确),忽略了模型对程序语义推理的能力。本文提出CORE,一个高质量、人工验证的基准,用于评估大模型在基础静态分析任务上的表现。CORE包含12,553个任务实例,覆盖C/C++、Java和Python程序中的数据依赖、控制依赖和信息流。为确保语义多样性和推理复杂性,我们提出一种基于结构覆盖和依赖深度的语义感知采样策略。我们评估了10个主流大模型,结果显示它们虽能较好识别依赖关系,但在需要深层语义理解和多步推理的任务上仍表现不足。通过定性分析,我们揭示了复杂控制结构和反向依赖等关键挑战,为提升大模型代码推理能力提供方向。
原文摘要 · Abstract (English)
Large language models (LLMs) have been widely adopted across diverse domains of software engineering, such as code generation, program repair, and vulnerability detection. These applications require understanding beyond surface-level code patterns: value propagation, control flow, and interdependence between program elements. However, existing benchmarks primarily evaluate end-to-end outcomes, such as whether code is correctly repaired or generated, leaving the models' ability for program semantic reasoning underexplored. This work presents CORE, a high-quality, human-verified benchmark designed to evaluate LLMs on fundamental static analysis tasks. CORE includes 12,553 task instances spanning data dependency, control dependency, and information flow across programs written in C/C++, Java, and Python. To ensure semantic diversity and reasoning complexity, we propose a semantics-aware diverse sampling strategy that selects targets and task instances based on structural coverage and dependency depth. We evaluate 10 mainstream LLMs and show that, while they perform well at identifying dependencies, models still struggle with tasks that require deeper semantic understanding and multi-step reasoning. We further conduct qualitative analyses to uncover key challenges, such as complex control structures and backward dependency patterns, offering insights into improving LLMs' code reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。