arXiv:2606.03077cs.LGcs.AI2026-06

Libra提升智能体强化学习训练效率,解决资源分配不均问题。

Libra: Efficient Resource Management for Agentic RL Post-Training

  • 全局资源规划+弹性池实现跨阶段动态分配
  • 采用因果反馈队列,使吞吐量提升3倍、收敛快2.5倍
  • 适合大规模智能体训练系统优化,尤其长尾负载场景

强化学习已成为塑造大语言模型为智能体的标准后训练范式。在智能体强化学习中,回放阶段调用工具生成轨迹,导致长尾且非平稳的工作负载,带来两大资源管理挑战:一是少数轨迹主导回放时长;二是回放与训练存在显著计算模式、内存需求和序列长度敏感度的不对称性。随着策略演化,序列长度分布持续漂移,静态资源划分逐渐失效。本文提出Libra系统,通过两项核心机制应对:其一为全局资源规划器,联合优化回放与训练集群的GPU分配,利用弹性混合资源池实现轻量级、无阻塞的跨阶段工作节点重分配;其二为基于因果信号的多级反馈队列(C-MLFQ)调度器,根据工具返回结果而非脆弱的长度预测,将请求路由至异构回放桶。在48块A800 GPU上评估显示,Libra相较基线最高提升3.0倍吞吐量,奖励收敛速度最快达2.5倍。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has emerged as a standard post-training paradigm for shaping large language models (LLMs) into capable agents. In agentic RL, the rollout stage generates trajectories while invoking tools, producing long-tailed and non-stationary workloads that expose two fundamental challenges in resource management. First, due to the long-tail distribution, a small fraction of trajectories dominates rollout makespan. Second, rollout and training are subject to cross-stage imbalance, as they exhibit strong asymmetry in compute patterns, memory demands, and sensitivity to sequence length. Compounding this asymmetry, the sequence length distribution drifts continuously as the policy evolves, rendering any static resource split progressively suboptimal. We present Libra, a resource management system to address both challenges via two core mechanisms. The first is a global resource planner that jointly optimizes GPU allocation across rollout and training clusters. It leverages an elastic hybrid pool to enable lightweight, non-blocking worker reallocation between stages. The second is a causality-driven multi-level feedback queue (C-MLFQ) scheduler, which routes requests to heterogeneous rollout buckets based on causal signals derived from tool-return outcomes, rather than relying on fragile length predictions. Evaluated on 48 A800 GPUs, Libra achieves up to 3.0x higher throughput and converges up to 2.5x faster in reward compared to the baselines.

强化学习资源管理大模型智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。