构建持续学习代码代理评估基准，测试其长期适应能力。
SWE-Bench-CL: Continual Learning for Coding Agents
- 基于时间序列的GitHub问题数据集，模拟真实开发演化过程。
- 提出综合评估指标，量化知识迁移与遗忘程度。
- 适合研究持续学习、AI编程助手的开发者和研究人员。
大型语言模型在静态代码生成基准上表现优异，但真实软件开发是持续演进的过程。我们引入SWE-Bench-CL，一个基于OpenAI与普林斯顿NLP 2024年发布的真人验证数据集SWE-Bench Verified的持续学习新基准。通过将GitHub问题按时间顺序组织，反映代码库自然演化，该基准可直接评估代理积累经验、跨任务知识迁移及抵抗灾难性遗忘的能力。我们配套提供：(i) 任务间结构相似性与上下文敏感性的初步分析；(ii) 基于LangGraph的交互式评估框架，集成FAISS语义记忆模块；(iii) 一系列专用持续学习指标——包括平均准确率、遗忘度、正向/反向迁移、工具使用效率，以及广义复合持续学习得分与CL-F-beta得分，以捕捉稳定性与可塑性权衡。我们设计严谨实验协议，对比多种Python仓库中带记忆与不带记忆代理的表现。所有代码与数据公开于https://github.com/thomasjoshi/agents-never-forget，为社区提供可复现平台，推动更适应、更鲁棒的软件工程AI代理发展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved impressive results on static code-generation benchmarks, but real-world software development unfolds as a continuous stream of evolving issues, fixes, and feature requests. We introduce SWE-Bench-CL, a novel continual learning benchmark built on the human-verified SWE-Bench Verified dataset introduced by OpenAI and Princeton-NLP in 2024. By organizing GitHub issues into chronologically ordered sequences that reflect natural repository evolution, SWE-Bench-CL enables direct evaluation of an agent's ability to accumulate experience, transfer knowledge across tasks, and resist catastrophic forgetting. We complement the dataset with (i) a preliminary analysis of inter-task structural similarity and contextual sensitivity, (ii) an interactive LangGraph-based evaluation framework augmented with a FAISS-backed semantic memory module, and (iii) a suite of specialized continual learning metrics -- including average accuracy, forgetting, forward/backward transfer, tool-use efficiency, and a generalized Composite Continual Learning Score and CL-F-beta score -- to capture the stability-plasticity trade-off. We outline a rigorous experimental protocol comparing memory-enabled and memory-disabled agents across diverse Python repositories. All code and data are publicly available at https://github.com/thomasjoshi/agents-never-forget, providing the community with a reproducible platform for developing more adaptive and robust AI agents in software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。