评测大模型在真实代码库中的推理能力，发现整合能力是主要瓶颈。
From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level
- 通过执行驱动的变异框架生成真实逻辑状态，避免记忆干扰。
- 提出三项指标：读取负荷、模拟深度、集成宽度，精准量化推理能力。
- 揭示当前顶尖模型在跨文件整合上存在明显短板，适合研发人员参考。
随着大语言模型演变为自主智能体，评估其在大规模真实代码库中保持逻辑一致性的能力变得至关重要。现有基准多局限于孤立代码片段或黑盒测试。我们提出RepoReason，一种基于归纳断言验证的白盒诊断基准。为消除记忆干扰并保留真实逻辑深度，采用执行驱动的变异框架，利用环境作为语义原理解析生成真实状态。同时构建细粒度诊断系统，结合动态程序切片，以三项正交指标量化推理能力：ESV（读取负荷）、MCL（模拟深度）和DFI（集成宽度）。对前沿模型（如Claude-4.5-Sonnet、DeepSeek-v3.1-Terminus）的全面评估显示，普遍存在整合缺陷，其中集成宽度是主要认知瓶颈。研究结果为优化下一代智能软件工程提供了细粒度白盒洞察。
原文摘要 · Abstract (English)
As large language models (LLMs) evolve into autonomous agents, evaluating repository-level reasoning, the ability to maintain logical consistency across massive, real-world, interdependent file systems, has become critical. Current benchmarks typically fluctuate between isolated code snippets and black-box evaluations. We present RepoReason, a white-box diagnostic benchmark centered on abductive assertion verification. To eliminate memorization while preserving authentic logical depth, we implement an execution-driven mutation framework that utilizes the environment as a semantic oracle to regenerate ground-truth states. Furthermore, we establish a fine-grained diagnostic system using dynamic program slicing, quantifying reasoning via three orthogonal metrics: $ESV$ (reading load), $MCL$ (simulation depth), and $DFI$ (integration width). Comprehensive evaluations of frontier models (e.g., Claude-4.5-Sonnet, DeepSeek-v3.1-Terminus) reveal a prevalent aggregation deficit, where integration width serves as the primary cognitive bottleneck. Our findings provide granular white-box insights for optimizing the next generation of agentic software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。