arXiv:2604.09107cs.DCcs.AI2026-04被引 3

提出可弹性扩展的模型权重传输系统,显著降低大模型强化学习训练延迟。

TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training

  • 基于参考存储抽象,按需调用已驻留权重,避免物理拷贝。
  • 在跨数据中心场景下,权重传输延迟降低19倍,GPU空等时间减少6.7倍。
  • 支持动态扩缩容,适用于多种推理负载,部署无需额外工程调整。

现代大语言模型强化学习(LLM RL)任务需要高效权重传输系统以实现异构资源上的训练扩展。现有方法或缺乏动态集群伸缩灵活性,或存在固有数据移动开销,导致性能低下。本文提出参考导向存储(ROS),一种针对强化学习权重传输的新存储抽象,利用模型权重在计算节点间的高度重复特性。ROS呈现特定版本权重可被按需获取的假象,实际并不物理存储任何副本,而是追踪持有这些权重的GPU工作节点。请求时直接调用其进行读取服务。我们构建了生产级系统TensorHub,进一步集成拓扑优化传输、强一致性与容错机制。评估显示,TensorHub能完全饱和RDMA带宽,并在三种不同回滚工作负载下仅需极少工程投入即实现自适应。具体而言,独立回滚场景中总GPU空等时间减少6.7倍;弹性回滚中权重更新加速4.8倍;跨数据中心回滚的空等时间下降19倍。该系统已在生产环境部署,支撑前沿强化学习训练。

原文摘要 · Abstract (English)

Modern LLM reinforcement learning (RL) workloads require a highly efficient weight transfer system to scale training across heterogeneous computational resources. However, existing weight transfer approaches either fail to provide flexibility for dynamically scaling clusters or incur fundamental data movement overhead, resulting in poor performance. We introduce Reference-Oriented Storage (ROS), a new storage abstraction for RL weight transfer that exploits the highly replicated model weights in place. ROS presents the illusion that certain versions of the model weights are stored and can be fetched on demand. Underneath, ROS does not physically store any copies of the weights; instead, it tracks the workers that hold these weights on GPUs for inference. Upon request, ROS directly uses them to serve reads. We build TensorHub, a production-quality system that extends the ROS idea with topology-optimized transfer, strong consistency, and fault tolerance. Evaluation shows that TensorHub fully saturates RDMA bandwidth and adapts to three distinct rollout workloads with minimal engineering effort. Specifically, TensorHub reduces total GPU stall time by up to 6.7x for standalone rollouts, accelerates weight update for elastic rollout by 4.8x, and cuts cross-datacenter rollout stall time by 19x. TensorHub has been deployed in production to support cutting-edge RL training.

大模型训练权重传输强化学习分布式系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。