拆解代码智能的细粒度评估框架,揭示模型能力并非全能。
CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks
- 将代码任务分解为原子级子任务,分离不同认知需求
- 78.55%的任务有效率,远超SWE-bench-Verified的31.7%
- 适合研究模型在代码理解、生成等维度的真实短板
现有大模型代码评估多依赖粗粒度通过率,将编程能力视为单一整体,掩盖了具体认知瓶颈。且静态基准易受数据污染和性能饱和影响。为此,我们提出CoreCodeBench,一个可配置的细粒度仓库级评估框架。通过自动化工具CorePipe,从Python仓库中提取并转化出一系列隔离特定认知需求的任务。相比静态评估,该框架支持可控难度调节,避免性能饱和,并保障数据质量。其任务有效性达78.55%,显著高于SWE-bench-Verified的31.7%。对主流LLM的实验显示,模型在不同认知维度排名差异显著,表明代码能力非单一,强项不必然代表全面优势。这凸显细粒度分类在诊断模型缺陷中的必要性,也为持续演进代码智能提供了可靠框架。核心工具代码开源于https://github.com/AGI-Eval-Official/CoreCodeBench,数据集可在https://huggingface.co/collections/tubehhh/corecodebench-68256d2faabf4b1610a08caa获取。
原文摘要 · Abstract (English)
The evaluation of Large Language Models (LLMs) for software engineering has shifted towards complex, repository-level tasks. However, existing benchmarks predominantly rely on coarse-grained pass rates that treat programming proficiency as a monolithic capability, obscuring specific cognitive bottlenecks. Furthermore, the static nature of these benchmarks renders them vulnerable to data contamination and performance saturation. To address these limitations, we introduce CoreCodeBench, a configurable repository-level benchmark designed to dissect coding capabilities through atomized tasks. Leveraging our automated framework, CorePipe, we extract and transform Python repositories into a comprehensive suite of tasks that isolate distinct cognitive demands within identical code contexts. Unlike static evaluations, CoreCodeBench supports controllable difficulty scaling to prevent saturation and ensures superior data quality. It achieves a 78.55% validity yield, significantly surpassing the 31.7% retention rate of SWE-bench-Verified. Extensive experiments with state-of-the-art LLMs reveal a significant capability misalignment, evidenced by distinct ranking shifts across cognitive dimensions. This indicates that coding proficiency is non-monolithic, as strength in one aspect does not necessarily translate to others. These findings underscore the necessity of our fine-grained taxonomy in diagnosing model deficiencies and offer a sustainable, rigorous framework for evolving code intelligence. The code for CorePipe is available at https://github.com/AGI-Eval-Official/CoreCodeBench, and the data for CoreCodeBench can be accessed at https://huggingface.co/collections/tubehhh/corecodebench-68256d2faabf4b1610a08caa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。