研究发现代码技能会无声过期,现有智能体难以有效更新。
Repo2Skill-Evo: Repository Skills Go Stale in Silence

- 将版本迭代视为技能维护任务,自动更新过时代码知识
- 57个真实仓库中90%以上版本升级导致技能部分失效
- 顶尖智能体平均只能正确维护30%-70%的技能内容
大型语言模型代理在不断演化的代码仓库中运行,其表现依赖于特定仓库的程序性知识:调用哪些API、执行哪些脚本、遵循当前版本的哪些规范。代理技能将此类知识外化为可复用单元,已有研究证明其能提升代理性能。但这种提升是否持久仍不明确。正是版本特异性使技能有用,也使其脆弱:版本发布后,技能可能悄然过期却无任何显式信号,仍提供过时指导。将知识外化为技能反而掩盖了其退化过程。本文研究代理能否保持外部化知识的时效性。Repo2Skill-Evo将每个版本过渡视为技能维护任务:给定V1技能集和官方的V1到V2补丁,代理需更新过时内容同时保留仍有效的指导。在57个真实仓库和105个选定版本过渡中,每个评估过渡均导致部分V1技能集失效。然而,六种前沿代理在基于补丁的移除指标下,平均@3宏F1仅为29.9%-69.7%,该指标平衡了过时内容召回率与过度编辑精确率。实验显示两种主要错误:技能集未覆盖受影响文件导致过时内容残留;过度编辑虽提高召回率但降低精确率。代码技能在静默中过期,即使前沿代理也无法可靠维护。
原文摘要 · Abstract (English)
Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。