用强化学习训练检索代理,让编程大模型少找文件、多写代码。
CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

- 用强化学习训练检索代理,同时并发执行grep、glob等命令找文件。
- 在SWE-Bench上减少19%的令牌消耗,解决率仍达27.0%。
- 适合希望提升编程大模型效率的研究者与开发者。
当前如Claude Code和OpenHands等大模型编程代理存在效率瓶颈:大量计算资源用于查找待修改文件而非实际编码。在SWE-Bench Verified数据集上,30B规模的OpenHands代理平均需23轮、63.1万令牌才能解决一个任务,其中大量操作集中在grep、glob和view_file等文件搜索步骤。本文提出CodeGrep,一个14B参数的检索代理,通过端到端的GRPO强化学习训练,能并行发起多轮grep、glob和read调用,返回候选文件给下游冻结的编程代理。在全部500个SWE-Bench Verified实例上,CodeGrep保持27.0%的解决率(基准为25.8%),同时减少15%的交互轮次和19%的令牌消耗。研究发现,检索器的下游效用取决于精度阈值:当精度低于0.375时(如BM25)会降低性能,0.445(如Jina)无影响,而CodeGrep达到0.677的精度后开始显著降低推理成本。为支持该研究,我们基于67K开源代理轨迹,使用CATM挖掘监督信号,并构建了支持多轮强化学习的Git-worktree环境。结果显示,在优势层而非奖励层施加效率信号,可减少KL漂移,更有效地转化为下游效率提升。相关模型、训练流程、强化学习环境与评估工具将公开发布。
原文摘要 · Abstract (English)
Modern LLM coding agents such as Claude Code and OpenHands share a common inefficiency: they spend much of their token budget finding the file to patch, rather than patching it. On SWE-Bench Verified, a 30B OpenHands agent averages 23 rounds and 631K tokens per resolved issue, with many calls spent on grep, glob, and view_file during repository exploration. We introduce CodeGrep, a 14B retrieval agent trained end-to-end with GRPO to issue multi-turn parallel grep, glob, and read tool calls and return candidate files to a frozen downstream coding agent. On all 500 SWE-Bench Verified instances, CodeGrep preserves resolve rate while substantially improving efficiency: 27.0% versus 25.8% for the no-retrieval baseline, with 15% fewer rounds and 19% fewer tokens on resolved instances. Across retrievers, downstream utility follows a precision threshold: BM25 with precision 0.375 degrades the agent, Jina with precision 0.445 is neutral, and CodeGrep with precision 0.677 crosses the threshold at which retrieval begins to reduce rollout cost. To enable this study, we mine supervision from 67K open-source agent trajectories using CATM and build a Git-worktree environment for multi-turn agent RL. In our setting, applying the efficiency signal at the advantage layer rather than the reward layer reduces KL drift and translates cleanly into downstream efficiency. We will release the model, training pipeline, RL environment, and evaluation harnesses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。