arXiv:2608.17393cs.AI2026-08

让代码智能体在真实环境中训练更稳定,突破执行与学习不一致的瓶颈。

LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents

论文配图:LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
图 1 · 摘自论文原文
  • 用进程内大模型代理捕获生成流,实现令牌级对齐与概率重计算。
  • 通过镜像缓存和分阶段防护,显著降低奖励作弊风险,提升执行可靠性。
  • 内置插件与实时界面,可监控训练过程并诊断行为轨迹,适合研究者使用。

代码智能体的强化学习日益依赖长期运行的代理框架来管理工具集成、代码库上下文和执行反馈。然而,这些框架的原生执行环境与策略梯度训练存在根本性错位:环境崩溃和奖励作弊会污染结果信号,而训练-推理差异使回放行为与策略更新脱节。为此,我们提出 LEGO-RL 框架,无需修改内部控制流即可将原生代码智能体代理与可扩展的策略梯度优化相衔接。该框架基于三大支柱:(1)通过进程内 LLM 代理实现忠实优化,捕捉原始生成流以进行令牌级对齐,并支持训练端对日志概率的鲁棒重计算,即使在代理端压缩或重新序列化后依然有效;(2)通过可扩展沙箱编排实现可靠执行,包含镜像缓存与分阶段防御机制,有效缓解奖励作弊;(3)通过集成插件自动验证与监控,并搭配实时可视化界面,实现细粒度轨迹诊断。我们在三个原生代码智能体代理上评估 LEGO-RL,使用 GSPO 训练 Qwen3.5-35B-A3B 模型,在 SWE-bench Verified 上分别将其性能从 64.0% 提升至 70.4%(OpenHands SDK)、62.4% 提升至 68.2%(Claude Code)和 57.2% 提升至 66.6%(OpenCode),同时保持回放-训练概率相关性高于 0.99。

原文摘要 · Abstract (English)

Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.

强化学习代码生成大模型训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。