arXiv:2507.22789cs.LGcs.AI2025-07被引 3

G-Core提升强化学习人类反馈训练效率,支持大规模多模态模型部署。

G-Core: A Simple, Scalable and Balanced RLHF Trainer

  • 采用并行控制器架构,避免单点瓶颈,灵活调度复杂训练流程。
  • 动态资源分配策略降低硬件空闲率,实测在波动负载下利用率显著提升。
  • 已在微信产品线成功落地,适合需高可用、可扩展的工业级大模型训练场景。

强化学习从人类反馈(RLHF)已成为训练大语言模型和扩散模型的主流范式。现有系统在扩展至多模态与扩散工作流、应对动态负载时面临挑战,尤其体现在控制器可扩展性、资源灵活部署和高效编排方面的局限。本文提出G-Core——一个简洁、可扩展且平衡的RLHF训练框架。G-Core引入并行控制器编程模型,实现复杂RLHF流程的灵活高效编排,突破单个中心控制器的瓶颈。同时,提出动态资源放置策略,自适应划分资源并调度任务,在高度变化的训练条件下显著降低硬件空闲时间,提升利用率。G-Core已成功训练支持微信产品功能的大规模模型,服务海量用户,验证了其在真实场景中的有效性与鲁棒性。实验表明,G-Core推动了RLHF训练的前沿水平,为未来大规模人对齐模型的研究与部署奠定坚实基础。

原文摘要 · Abstract (English)

Reinforcement Learning from Human Feedback (RLHF) has become an increasingly popular paradigm for training large language models (LLMs) and diffusion models. While existing RLHF training systems have enabled significant progress, they often face challenges in scaling to multi-modal and diffusion workflows and adapting to dynamic workloads. In particular, current approaches may encounter limitations in controller scalability, flexible resource placement, and efficient orchestration when handling complex RLHF pipelines, especially in scenarios involving dynamic sampling or generative reward modeling. In this paper, we present \textbf{G-Core}, a simple, scalable, and balanced RLHF training framework designed to address these challenges. G-Core introduces a parallel controller programming model, enabling flexible and efficient orchestration of complex RLHF workflows without the bottlenecks of a single centralized controller. Furthermore, we propose a dynamic placement schema that adaptively partitions resources and schedules workloads, significantly reducing hardware idle time and improving utilization, even under highly variable training conditions. G-Core has successfully trained models that support WeChat product features serving a large-scale user base, demonstrating its effectiveness and robustness in real-world scenarios. Our results show that G-Core advances the state of the art in RLHF training, providing a solid foundation for future research and deployment of large-scale, human-aligned models.

RLHF分布式训练资源调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。