arXiv:2605.26177cs.SEcs.AI2026-05被引 3

通过扰动测试发现代码智能体缺乏跨文件上下文推理能力

RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations

论文配图:RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
图 1 · 摘自论文原文
  • 用三类语义保持的仓库扰动增强上下文推理需求
  • 任务准确率从66.8%降至25.3%,暴露结构理解短板
  • 提出结构优先框架,提升代码智能体上下文建模能力

当前代码智能体在仓库级软件工程基准上表现优异,但其在端到端任务(如问题修复)中的成功是否真正反映了对仓库上下文的推理能力仍不明确。为此,我们提出RepoMirage,一个基于SWE-Bench Verified的两阶段评估套件,利用扰动作为诊断工具,通过改变仓库呈现方式来强化上下文推理需求。RepoMirage-Perturb引入三种语义保持的仓库级扰动,揭示当正确求解需要更广范围上下文访问时性能明显下降。RepoMirage-Extend进一步将扰动导致的结构性瓶颈转化为显式任务,平均准确率从原设置的66.8%降至25.3%,表明代码智能体在仓库上下文推理方面存在显著缺陷。轨迹分析显示,智能体虽能访问更广上下文,却未能有效转化为结构信息。基于此,我们提出RepoAnchor——一种结构优先的原型工作流,将仓库探索与下游求解分离,实验表明显式结构引导可带来显著提升。这些结果揭示了代码智能体在仓库上下文推理上的未被重视的差距,并表明更强的结构感知方法有望改善其表现。

原文摘要 · Abstract (English)

Code agents are currently having skillful performance on repository-level software engineering benchmarks, but it remains unclear whether success on end-to-end tasks such as issue resolution truly reflects repository context reasoning, the ability to identify the task-relevant information across multiple files and reason over the relations among them. To investigate this question, we introduce RepoMirage, a two-stage evaluation suite built on SWE-Bench Verified that adopts perturbation as a diagnostic tool to increase the demand for context reasoning by transforming how the repository is exposed. First, RepoMirage-Perturb applies three types of semantics-preserving repository-level perturbations, revealing a clear performance drop when correct solving requires broader context access. RepoMirage-Extend further turns perturbation-targeted structural bottlenecks into explicit tasks beyond issue resolution, where the average performance declines from 66.8% in the original setting to 25.3%, indicating a significant deficiency in repository context reasoning. Further trajectory analysis reveals an exploration drift, where agents access broader repository context but fail to turn it into effective structure information. Motivated by this observation, we propose RepoAnchor, a structure-first prototype workflow that separates repository exploration from downstream problem solving, and show that explicit structural scaffolding yields notable gains. These results uncover an previously overlooked gap in repository context reasoning for code agents and suggest that stronger structure-aware methods are potential to improve them.

代码生成上下文推理智能体评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。