测试AI生成代码的可维护性，发现多数代码虽能运行却难维护。
Needle in the Repo: A Benchmark for Maintainability in AI-Generated Repository Edits
- 设计小型多文件项目，用诊断探针测试代码可维护性维度。
- 平均仅36.2%的代码修改保持结构良好，最高认知为57.1%。
- 适合关注代码长期可维护性的开发者与评测人员。
AI编程代理已能完成复杂任务，但现有评估多聚焦行为正确性，忽视模块化、可测试性等可维护性风险。我们提出Needle in the Repo (NITR)，一种诊断式探针-验证框架，用于评估行为正确代码是否保持可维护结构。NITR将常见软件工程原则转化为嵌入小规模真实多文件代码库的可控探针，每个探针仅依赖单一可维护性维度。每个探针配以隐藏评估工具，结合功能测试与结构断言，返回可解释诊断结果。使用NITR，我们在GPT、Claude、Gemini和Qwen系列中评估23种编码配置，涵盖直接推理与代理模式。当前系统仍不稳健：平均仅36.2%案例成功，最佳为57.1%，多步任务性能从微任务的53.5%降至20.6%。最难点在于架构层面，如依赖控制（4.3%）与职责分解（15.2%）。此外，64/483结果（13.3%）通过功能测试但未通过结构断言。代理模式使平均性能从28.2%提升至45.0%，但未能消除架构缺陷。结果表明，代码生成进展不等于可维护演化进步，且NITR揭示了传统评估忽略的关键失败面。
原文摘要 · Abstract (English)
AI coding agents can now complete complex programming tasks, but existing evaluations largely emphasize behavioral correctness and often overlook maintainability risks such as weak modularity or testability. We present Needle in the Repo (NITR), a diagnostic probe-and-oracle framework for evaluating whether behaviorally correct repository edits preserve maintainable structure. NITR distills recurring software engineering wisdom into controlled probes embedded in small, realistic multi-file codebases, each designed so that success depends primarily on one targeted maintainability dimension. Each probe is paired with a hidden evaluation harness that combines functional tests for required behavior with structural oracles that encode the targeted maintainability constraint and return interpretable diagnoses. Using NITR, we evaluate 23 coding configurations across GPT, Claude, Gemini, and Qwen families in both direct-inference and agent-based settings. Current AI coding systems remain far from robust: on average, configurations solve only 36.2% of cases, the best reaches 57.1%, and performance drops from 53.5% on micro cases to 20.6% on multi-step cases. The hardest pressures are architectural rather than local edits, especially dependency control (4.3%) and responsibility decomposition (15.2%). Moreover, 64/483 outcomes (13.3%) pass all functional tests yet fail the structural oracle. Under our harness, agent-mode configurations improve average performance from 28.2% to 45.0%, but do not eliminate these architectural failures. These results show that progress in code generation is not yet progress in maintainable code evolution, and that NITR exposes a critical failure surface missed by conventional evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。