编码代理真正需要的上下文极少,核心代码本身已足够。
What Context Does a Coding Agent Actually Need to Act?
- 只保留待编辑代码本身,上下文压缩至原大小的1/3
- 自然语言摘要几乎无法替代原始代码,准确率仅4/45
- 模型规模差异不影响摘要效果,最小上下文即可解决问题
现代编码代理可承载整个代码库于上下文窗口中,但多数读取是浪费的。真正关键的问题不是能用多少上下文,而是实际需要多少。我们聚焦于代理必须修改代码的时刻,将定位工作与执行分离,固定定位由理想器完成,仅改变代码表示方式,并在SWE-bench Verified上评估真实问题解决率。结果表明,所需上下文极为有限:待编辑代码本身的信息已足够,其自然语言摘要对行为问题的解答能力远低于源代码(4/45 对比 27/45,独立评委评估,未见数据集);且该差距属于表示方式问题,而非摘要模型性能——前沿模型与30亿参数模型表现无异。周围上下文也几乎无影响:在所有多文件实例中,无论将文件其余部分渲染为UML骨架或签名,还是直接删除,解决率均无差异(N=70,McNemar检验p=0.75)。压缩上下文后,仅需19K上下文标记即可解决一个任务,远低于原需94K。此外,温度为0的API推理在字节相同的运行间导致约9%的实例结果翻转,构成基准上的噪声底限。我们发布该工具:经过验证的环境、每例参考修改的可表达性证明、确定性补丁生成及预注册假设的零假设公开。
原文摘要 · Abstract (English)
A modern coding agent can hold an entire repository in its context window. Most of its reading is wasted -- and the interesting question is not how much context an agent can use, but what it actually \emph{needs}. We study that question at the moment it matters most: when the agent must \emph{edit} code. Separating \emph{finding} the work site from \emph{acting} on it, we hold localization fixed with an oracle, vary only how the code is represented, and score context against real issue resolution on SWE-bench Verified. The answer is starkly minimal. The signal lives in the code being edited itself: natural-language summaries of it answer almost none of the behavioral questions that the source answers ($4/45$ vs.\ $27/45$, held-out repositories, independent judge), and the gap belongs to the representation, not the summarizer -- a frontier model's summaries score exactly as poorly as a 3B model's. The surrounding context hardly matters either: across every multi-file instance in Verified, under a protocol frozen before any data, rendering a file's remainder as UML skeletons and signatures resolves no more issues than deleting that remainder outright ($N{=}70$, exact McNemar $p{=}0.75$). That was our registered hypothesis, and it failed. Compressed context, meanwhile, matches whole files at a third of the tokens: a resolved issue costs $19$K context tokens, not $94$K. The instrument also yielded a finding the field should keep: temperature-0 API inference flips ${\sim}9\%$ of per-instance outcomes between byte-identical runs. That is a noise floor under every small effect reported on this benchmark, including ours. We release the instrument -- gold-validated environments, per-instance proof that every reference edit is expressible from every arm's context, deterministic patch construction, and pre-registered hypotheses whose nulls we publish.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。