arXiv:2606.09138cs.LGcs.CL2026-06被引 1

为智能体强化学习构建可管理的步骤级数据中间件,提升数据利用率。

Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning

论文配图:Claw-R1: A Step-Level Data Middleware System for Agentic Reinforcement Learning
图 1 · 摘自论文原文
  • 通过网关与数据池实现多轮交互数据的统一采集与结构化存储。
  • 支持实时查看每一步的状态、动作和奖励,并按质量筛选数据。
  • 适合研究智能体训练数据管理的学者与开发交互式智能体的工程师。

智能体强化学习(Agentic RL)已成为将大模型从静态聊天机器人转变为交互式智能体的重要后训练范式,催生了OpenClaw等代表性应用。现有工作主要关注策略优化算法与训练框架,却较少关注智能体-环境交互中数据全生命周期的管理,包括从数据生成到训练消费的全过程。为弥补这一空白,我们提出Claw-R1,一个面向智能体强化学习的交互式步骤级数据中间件系统。Claw-R1通过网关服务器与数据池两个核心组件,连接异构的智能体运行时与强化学习训练后端:网关服务器通过统一的LLM API入口捕获多轮交互步骤,数据池则将这些步骤组织为包含提示ID、响应ID、奖励及其他元数据的步骤级记录。在演示中,用户可实时检查轨迹,查看每一步的状态、动作与奖励,根据质量和可用性对数据进行筛选,并为不同下游强化学习算法配置训练就绪的数据批次。总体而言,Claw-R1将智能体交互轨迹视为可管理的数据资产,而非临时的运行日志。通过此演示,我们希望推动社区认识到数据管理在智能体强化学习中的重要性。代码已开源至https://github.com/AgentR1/Claw-R1,演示视频见https://youtu.be/Pw47dAOw6B0。

原文摘要 · Abstract (English)

Agentic reinforcement learning (RL) has become an important post-training paradigm for turning LLMs from static chatbots into interactive agents, giving rise to representative applications such as OpenClaw. Existing work mainly focuses on policy optimization algorithms and training frameworks, but pays less attention to the full data lifecycle of agent-environment interactions, from data production to training consumption. To bridge this gap, we present Claw-R1, an interactive step-level data middleware system for agentic RL. Claw-R1 connects heterogeneous agent runtimes with RL training backends through two core components: a Gateway Server and a Data Pool. The Gateway Server captures multi-turn interaction steps through a unified LLM API entry point, while the Data Pool organizes them into step-level records consisting of prompt IDs, response IDs, rewards and other metadata. In our demo, users can interactively inspect live trajectories, examine the state, action, and reward of each step, curate data by quality and readiness, and configure training-ready batches for different downstream RL algorithms. Overall, Claw-R1 treats agent interaction traces as managed data assets rather than temporary runtime logs. Through this demonstration, we hope to encourage the community to recognize the importance of data management in agentic RL. Our code is available at https://github.com/AgentR1/Claw-R1 and the demonstration video can be found at link https://youtu.be/Pw47dAOw6B0.

强化学习智能体数据管理LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。