构建合成环境提升代码代理跨任务泛化能力
Hybrid-Gym: Training Coding Agents to Generalize Across Tasks
- 分解代码操作轨迹，提炼可迁移技能并设计辅助训练任务
- 在SWE-Bench Verified上提升25.4%，显著优于基线模型
- 适合希望提升代码生成泛化能力的研究者与开发者
评估代码代理质量的主流基准多聚焦于GitHub上的单一问题求解，如SWE-Bench。然而在实际应用中，代理需完成更多样且复杂的任务，涉及代码库探索、软件测试与架构设计等多重技能。本文首次通过将行为轨迹细粒度分解，识别出跨任务共享的可迁移技能，并据此提出一套辅助训练任务的设计原则。基于此，我们构建了可扩展的合成任务环境Hybrid-Gym，包含函数定位、依赖搜索等任务。实验表明，经合成任务训练的代理在未见的真实任务上实现良好泛化，在SWE-Bench Verified上取得25.4%绝对提升，SWT-Bench Verified上7.9%，Commit-0 Lite上5.1%。Hybrid-Gym还能增强下游数据集（如SWE-Play）表现，使SWT-Bench Verified提升4.9%。代码已开源。
原文摘要 · Abstract (English)
When assessing the quality of coding agents, predominant benchmarks focus on solving single issues on GitHub, such as SWE-Bench. In contrast, in real use, these agents solve more various and complex tasks that involve other skills such as exploring codebases, testing software, and designing architecture. In this paper, we first characterize some transferable skills that are shared across diverse tasks by decomposing trajectories into fine-grained components, and derive a set of principles for designing auxiliary training tasks to teach language models these skills. Guided by these principles, we propose a training environment, Hybrid-Gym, consisting of a set of scalable synthetic tasks, such as function localization and dependency search. Experiments show that agents trained on our synthetic tasks effectively generalize to diverse real-world tasks that are not present in training, improving a base model by 25.4% absolute gain on SWE-Bench Verified, 7.9% on SWT-Bench Verified, and 5.1% on Commit-0 Lite. Hybrid-Gym also complements datasets built for the downstream tasks (e.g., improving SWE-Play by 4.9% on SWT-Bench Verified). Code available at: https://github.com/yiqingxyq/Hybrid-Gym.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。