arXiv:2603.13019cs.DCcs.AI2026-03

解决智能体强化学习的资源浪费问题,实现高效共享与弹性调度。

ARL-Tangram: Unleash the Resource Efficiency in Agentic Reinforcement Learning

  • 在动作级别统一管理外部资源,支持细粒度共享与动态调度。
  • 实测将任务完成时间缩短4.3倍,训练步骤提速1.5倍,节省71.2%外部资源。
  • 适合大规模智能体强化学习系统,尤其适用于云上复杂任务训练。

智能体强化学习(Agentic RL)作为云集群中的新型负载,使大语言模型通过与真实世界交互解决复杂问题。然而,与传统强化学习不同,其需要大量外部云资源,如用于代码执行的CPU和用于奖励模型的GPU,这些资源位于主训练集群之外。现有框架通常采用静态过度分配,即资源长期绑定于轨迹或隔离于任务,导致严重资源浪费。本文提出动作级别编排机制,并集成至ARL-Tangram——一个统一的资源管理系统,支持细粒度外部资源共享与弹性扩展。ARL-Tangram采用统一的动作级建模与弹性调度算法,在满足异构资源约束的前提下最小化动作完成时间(ACT)。此外,针对异构特性与拓扑结构,设计专用资源管理器以高效支持动作级执行。在真实世界智能体强化学习任务上的评估表明,ARL-Tangram可将平均ACT提升4.3倍,加速训练步长达1.5倍,并节省高达71.2%的外部资源。该系统已部署用于MiMo系列模型的训练。

原文摘要 · Abstract (English)

Agentic reinforcement learning (RL) has emerged as a transformative workload in cloud clusters, enabling large language models (LLMs) to solve complex problems through interactions with real world. However, unlike traditional RL, agentic RL demands substantial external cloud resources, e.g., CPUs for code execution and GPUs for reward models, that exist outside the primary training cluster. Existing agentic RL framework typically rely on static over-provisioning, i.e., resources are often tied to long-lived trajectories or isolated by tasks, which leads to severe resource inefficiency. We propose the action-level orchestration, and incorporate it into ARL-Tangram, a unified resource management system that enables fine-grained external resource sharing and elasticity. ARL-Tangram utilizes a unified action-level formulation and an elastic scheduling algorithm to minimize action completion time (ACT) while satisfying heterogeneous resource constraints. Further, heterogeneous resource managers are tailored to efficiently support the action-level execution on resources with heterogeneous characteristics and topologies. Evaluation on real-world agentic RL tasks demonstrates that ARL-Tangram improves average ACT by up to 4.3$\times$, speeds up the step duration of RL training by up to 1.5$\times$, and saves the external resources by up to 71.2$\%$. This system has been deployed to support the training of the MiMo series models.

强化学习资源调度云平台

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。