arXiv:2605.23945cs.AIcs.DC2026-05

动态调整并行策略,显著加速长尾生成的强化学习训练。

Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism

论文配图:Accelerating Long-Tail Generation in Synchronous RLHF Training via Adaptive Tensor Parallelism
图 1 · 摘自论文原文
  • 根据生成进度实时调整张量并行配置,避免资源浪费。
  • 相比原系统,生成延迟降低34.6%,整体训练迭代延迟减少27.2%。
  • 适合需要高效长文本生成的RLHF训练场景,尤其对大模型实用。

基于人类反馈的强化学习(RLHF)已成为提升模型质量的关键后训练范式。然而,同步三阶段RLHF流程常因生成阶段的响应长度偏差而受阻:解码过程中有效批大小迅速缩小,导致部分GPU闲置,少数长响应持续未完成。主流框架采用静态张量并行(TP)配置,无法适应动态批特征变化,性能潜力未被充分挖掘。本文提出PAT,一种在每轮RLHF生成阶段动态重配置TP的方法。PAT引入两项关键技术:首先,基于离线预分析的预测引导在线重配置机制,仅在预测延迟收益大于重配置开销时触发;其次,轻量级在线重配置机制仅更新受影响的状态与布局:通过代价模型选择KV缓存迁移或重计算以适配未完成解码状态,执行就地权重重分片,并复用已缓存通信组。我们在SGLang上实现PAT,并集成至VeRL框架。在LLaMA3.1-8B和Qwen3-14B上使用DeepScaleR评估表明,PAT相较原VeRL设置,生成延迟最高降低34.6%,端到端训练迭代延迟最高降低27.2%。

原文摘要 · Abstract (English)

Reinforcement Learning from Human Feedback (RLHF) has become a key post-training paradigm for improving model quality. However, the synchronous three-stage RLHF pipeline is often bottlenecked by the generation stage, where response-length skew causes the effective batch size to shrink rapidly during decoding, leaving GPUs underutilized while a few long responses remain unfinished. Mainstream frameworks employ a static tensor parallelism (TP) configuration that cannot adapt to changing batch characteristics, leaving substantial performance headroom unexplored. We propose PAT, an adaptive TP method that dynamically reconfigures TP during the generation stage of each RLHF iteration. PAT introduces two key techniques. First, a predictor-guided online reconfiguration method decides both the reconfiguration point and the target TP configuration based on offline profiling, triggering reconfiguration only when the predicted latency benefit outweighs the reconfiguration overhead. Second, a lightweight online reconfiguration mechanism updates only the states and layouts affected by TP changes: it adapts unfinished decoding states through a cost-model-based choice between KV-cache migration and recomputation, performs in-place weight resharding, and reuses cached communication groups. We implement PAT on top of SGLang and integrate it with the VeRL framework. Evaluations on LLaMA3.1-8B and Qwen3-14B using DeepScaleR show that PAT reduces generation latency by up to 34.6% and end-to-end RLHF training iteration latency by up to 27.2% compared to the original VeRL setup.

RLHF并行优化生成加速大模型训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。