arXiv:2601.07376cs.AIcs.DC2026-01被引 4

让多个LLM代理共享资源训练,同时保持策略更新独立可控。

OpenTinker: Separating Concerns in Agentic Reinforcement Learning

  • 用LoRA适配器管理动态策略状态,支持多任务并行训练。
  • 同一数据路径兼容监督微调与强化学习,轨迹转为带掩码的文本序列。
  • 适合需要多代理协作或高效资源复用的研究者使用。

我们提出 extsc{OpenTinker},一个开放的基础设施,用于在共享执行资源下训练大型语言模型(LLM)代理,支持多个基于LoRA的策略。现代代理工作负载包含监督微调(SFT)、在线强化学习(RL)、回放生成、验证及多轮环境交互。在此类任务中,LoRA适配器并非静态推理产物,而是频繁更新的策略状态,其优化器状态、回放快照和训练数据归属需保持一致。 extsc{OpenTinker} 将运行时围绕策略生命周期构建。用户定义环境、代理和学习目标,系统则管理训练客户端、回放采样器、检查点句柄和策略版本刷新。相同数据路径支持SFT与RL:将轨迹转换为带显式掩码的标记序列,上下文与环境观测条件模型,生成的动作标记携带监督权重或强化学习优势。该设计实现多LoRA的SFT/RL训练,允许多用户、多任务或代理共享基础模型,同时隔离适配器更新、检查点与回放快照。我们描述系统架构、适配器生命周期、当前实现中的服务支撑快照移交机制、混合适配器回放内核的后端契约,以及隔离适配器本地梯度与优化器状态的训练调度器。代表性验证任务涵盖单轮、多轮、LoRA及多智能体代理训练。

原文摘要 · Abstract (English)

We introduce \textsc{OpenTinker}, an open infrastructure for training large language model (LLM) agents with many LoRA-backed policies over shared execution resources. Modern agent workloads mix supervised fine-tuning (SFT), online reinforcement learning (RL), rollout generation, validation, and multi-turn environment interaction. In such workloads, LoRA adapters are not static inference artifacts: they are frequently updated policy states whose optimizer state, rollout snapshot, and training data attribution must remain consistent. \textsc{OpenTinker} centers the runtime around this policy lifecycle. Users define environments, agents, and learning objectives, while the system manages training clients, rollout samplers, checkpoint handles, and policy-version refresh. The same data path supports SFT and RL by converting trajectories into token sequences with explicit masks: context and environment observations condition the model, while generated action tokens carry supervised weights or RL advantages. This design enables multi-LoRA SFT/RL training in which many users, tasks, or agents can share a base model while keeping adapter updates, checkpoints, and rollout snapshots isolated. We describe the system architecture, the adapter lifecycle, the service-backed snapshot handoff used by the current implementation, the backend contract for mixed-adapter rollout kernels, and the training scheduler that isolates adapter-local gradients and optimizer state. Representative validation tasks exercise single-turn, multi-turn, LoRA, and multi-agent agentic training.

强化学习LLM代理LoRA多任务训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。