arXiv:2608.17528cs.AIcs.SE2026-08

轻量框架实现可控智能体强化学习,6千样本提升代码能力14.6点

Agent Lightning v1.0: Towards Harnessed Agentic RL

论文配图:Agent Lightning v1.0: Towards Harnessed Agentic RL
图 1 · 摘自论文原文
  • 通过解耦架构让任意智能体接入强化学习训练
  • 仅用6000样本使代码模型在SWE-bench上准确率提升至56.4%
  • 开源完整流程,支持可复现的智能体强化学习研究

现代智能体依赖智能体约束框架管理工具、上下文和控制流,使约束成为系统关键部分。本文提出Agent Lightning v1.0,一个约3500行代码的轻量级框架,支持任意智能体约束接入强化学习训练。该范式称为受控智能体强化学习,其部署时的约束直接参与模型后训练,训练器仅观察LLM请求-响应序列。这带来重分词、样本合并、优势计算、损失归一化和后端调度等挑战,影响训练稳定性与效果。我们在指令遵循、搜索和编码智能体上评估该框架,提供完整的编码智能体强化学习可复现流水线。仅用6000个训练样本和适度算力,即实现Qwen3.5-9B在SWE-bench Verified上的准确率从41.8%提升至56.4%,绝对提升14.6个百分点。项目代码与训练脚本已公开,推动受控智能体强化学习的可复现研究。

原文摘要 · Abstract (English)

Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.

智能体强化学习代码生成可复现

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。