研究代码代理在大型项目中保持上下文一致性时的失效机制。
The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks
- 用耦合事实图建模代码编辑中的上下文依赖,区分近期上下文与参数记忆。
- 缺失关键事实会导致错误生成而非不执行,且距离不影响修复效果。
- 适合关注代码生成可靠性、模型上下文管理能力的研究者。
大型代码库任务要求代理在有限上下文窗口内维持测试、导入、配置和迁移规则的一致性。本文将其建模为重构耦合事实图:每次编辑所需的事实来自近期上下文或参数记忆,两者均未覆盖则形成一致性债务。通过七种模型和五个测试框架,在不同通道供应与剥夺条件下注入故障。结果表明,当两个通道均无事实时,所有模型无法完成未见过API的任务;将事实放入提示后成功率恢复。当重命名破坏模型对真实库的记忆时,七种模型在同一位置全部失败,通过与未通过的测试一致。事实可用性决定成败,距离无关:缺漏一个事实恰好导致等量工作损失,提供事实无论远近皆有效。不同测试框架消耗令牌差异超十倍,因重复重建相同内容速率不同,多耗无益。缺失事实产生错误行为而非空缺,代理会主动伪造文件或猜测值,因此基于读取构建的检测工具会误判为已填洞。代理声称受阻的比例是模型固有属性,从每次试验都报告到全不报告。可用性不能解决所有问题:当标准与代码冲突时,代理仍遵循标准,即使其推荐更差代码,故过时规范文件代价高于无文件。因参数记忆可替代读取,在SWE-bench上模型可能已知仓库,读取不再预测成功。建议测试框架应确保编辑依赖的事实在代理写入时可用,并根据代理产出而非读取判断可用性。
原文摘要 · Abstract (English)
Repository-scale coding requires an agent to keep tests, imports, configuration, and migration rules consistent within a bounded context window. We model this as reconstructing a coupled-fact graph: at each edit, a required fact comes from recent context or parametric memory, and the facts covered by neither form coherence debt. We supply and withhold each channel and inject faults across seven models and five harnesses. As expected, no model completes a task on an unseen API with both channels empty, and putting the facts in the prompt restores success. When a rename defeats what models memorized about a real library, all seven fail in the same place, passing and missing the same tests. Availability decides the outcome and distance does not: withholding a fact costs exactly the work it supports, and a supplied fact works as well far from the edit as next to it. Harnesses pay unequal prices for it: configurations that all pass every test differ more than tenfold in tokens consumed because they rebuild the same content at different rates, and spending more recovers nothing when facts are withheld. A missing fact produces wrong work rather than absent work: an agent asked to act acts, fabricating the file or guessing the value, so instruments built on reads look for a hole already filled. How often it says it is blocked instead is a property of the model, from every trial to none. Availability does not settle every edit: where standard and code disagree, agents follow the standard even when it prescribes the worse code, so a stale convention file costs more than no file. Because parametric memory substitutes for reading, on SWE-bench, where models likely know the repositories, reads no longer predict success. Harnesses should keep the facts an edit depends on available when the agent writes, and check that availability against what the agent produces rather than what it reads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。