实验证明代码助手的上下文文件对准确率无显著提升。
Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
- 对比两个前沿代码代理，系统性测试上下文文件效果
- 无论哪个代理，上下文文件均未提升正确率（≤10-15%）
- 失败主因是实现能力不足，非仓库知识缺失，适合研究者参考
持续性上下文文件（如 AGENTS.md、CLAUDE.md）是引导AI代码代理的标准做法，但其有效性证据矛盾。我们对两个前沿代理（Claude Code 和 Codex）在3个真实仓库中的17项任务（15项共享+2项Codex独有）进行了受控消融实验，共执行288次评估并采用黄金测试集评测。结果显示，上下文策略对任一代理的正确率均无显著影响（等效性检验下上限≤10-15%）。故障模式分析表明：代理失败主因在于实现能力——特征设计、模式选择与精确连接，而非上下文文件可提供的仓库知识；操控探针验证显示，真实的AGENTS.md无法将接近成功的尝试转为通过。此外，边缘难度任务表现具有代理特异性（斯皮尔曼等级相关系数=0.75），这可能解释了以往研究结果不一致的原因：单代理研究选取的任务分布不同。所有代码、数据与分析均已公开。
原文摘要 · Abstract (English)
Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。