评测编程大模型能否复用过往经验,提升代码任务解决效率。
SWE Context Bench: A Benchmark for Context Learning in Coding
- 设计1100个基础任务与376个关联任务,模拟真实代码库上下文关系。
- 正确复用历史经验可提升准确率,降低运行时间和令牌消耗。
- 适合研究代码生成模型上下文管理能力的开发者和研究人员。
大型语言模型越来越多地被用作软件工程任务的编码代理。现有基准主要评估代理是否能正确解决请求或修复缺陷,但大多将任务视为独立,未考察代理能否跨相关问题复用先前经验。因此,复用经验带来的效率提升难以衡量。本文提出SWE-ContextBench,一个专门用于评估编码代理上下文理解与检索能力的基准。该基准包含1,100个基础任务和376个由真实GitHub问题与拉取请求间依赖关系衍生的关联任务,覆盖51个独立仓库和9种编程语言。基准评估在已有上下文的情况下,代理解决相关问题的准确性和效率。通过该基准,我们研究了多种编码代理在不同上下文复用设置和检索策略下的表现。结果表明,准确总结并检索历史经验可显著提升解决准确率,并降低运行时间和令牌成本,尤其在较难任务上效果更明显。相反,未经筛选或错误选择的上下文带来有限甚至负面收益。这些发现凸显了上下文管理与检索准确性的重要性,确立了SWE-ContextBench作为研究编码代理上下文学习的可靠基准。
原文摘要 · Abstract (English)
Large language models are increasingly used as coding agents for software engineering tasks. Current benchmarks mainly evaluate whether the agent can correctly solve the request or fix the bugs. They largely treat tasks as independent and do not assess whether agents can reuse previous experience across related problems. As a result, the efficiency gains from reusing the previous experience remains difficult to measure. We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate context understanding and retrieval in coding agents. SWE-ContextBench consists of 1,100 base tasks with another 376 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests. SWE-ContextBench groups base tasks and related tasks with shared context across 51 unique repositories and 9 programming languages. The benchmark evaluates how accurately and efficiently agents solve related issues when prior cases are available in context. Using SWE-ContextBench, we study the behavior of multiple coding agents across varying context reuse settings and retrieval strategies. Our results show that accurately summarized and retrieved previous experience can significantly improve resolution accuracy and reduce runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected context provides limited or negative benefits. These findings highlight the importance of context management and retrieval accuracy, and position SWE-ContextBench as a principled benchmark for studying context learning in coding agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。