多租户异步强化学习服务，让多个用户共享算力高效训练智能体。
MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service
- 用轻量LoRA适配器共享基础模型，支持多用户并发微调。
- 异步解耦流水线，32个任务并行时加速比达4.3倍，训练时间降85%。
- 适合需要快速迭代智能体的科研或企业团队使用。
基于可验证奖励的强化学习（RLVR）显著提升了大语言模型在多轮代理任务中的推理能力，尤其在涉及工具使用的环境交互场景中表现优异。然而，由于计算开销巨大，模型微调成本过高，限制了其广泛应用。本文提出MARLaaS（多租户异步强化学习即服务），支持多个用户和任务并发进行强化学习微调。系统基于两个核心思想：(1) 通过轻量级LoRA适配器实现多租户共享基础模型；(2) 采用解耦的异步架构，将轨迹生成、环境交互与策略训练分离为独立调度阶段。该设计使各任务可按自身节奏在事件驱动下推进，减少任务间干扰、空闲时间和端到端延迟。在最多32个并发任务的多任务场景中，MARLaaS在保持单任务顶尖性能的同时，提升加速器利用率最高达4.3倍，端到端训练时间减少85%。
原文摘要 · Abstract (English)
Reinforcement Learning from Verifiable Rewards (RLVR) has significantly improved the reasoning capabilities of large language models (LLMs), particularly in multi-turn agentic settings involving environment interaction like tool use. However, fine-tuning such models remains prohibitively expensive due to high computational requirements, limiting accessibility. We propose MARLaaS (Multi-tenant Asynchronous RL as a Service), a system for concurrent RL fine-tuning across multiple users and tasks. Our approach is based on two key ideas: (1) sharing a base model across tenants using lightweight LoRA adapters, and (2) a disaggregated asynchronous architecture that decouples rollout generation, environment interaction, and policy training into independently scheduled stages. This design enables tasks to progress through the RL pipeline at their own pace in an event-driven manner, reducing cross-task interference, idle time, and end-to-end latency. In multi-task settings (we report up to 32 concurrent tasks), MARLaaS achieves single-task state-of-the-art performance while improving accelerator utilization by up to 4.3x and reducing end-to-end training time by 85%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。