剖析大模型强化学习训练中的系统瓶颈，提出真实负载评估工具。
RL in the Wild: Characterizing RLVR Training in LLM Deployment
- 分析不同训练阶段的负载分布与动态变化规律。
- 发现序列长度不均导致显卡空转，任务并行效率低下。
- 发布PolyTrace基准套件，实测准确率达94.7%，适合系统优化研究者。
大型语言模型（LLMs）已广泛应用于多个领域。随着快速发展，基于可验证奖励的强化学习（RLVR）在近几个月迅速兴起，以提升模型推理与理解能力。然而，其复杂的数据流和多样任务给强化学习训练系统带来巨大挑战，目前对RLVR的系统性认知仍有限。为深入理解RLVR引入的系统挑战，本文对实际部署中的RLVR任务进行了表征研究，重点分析不同训练阶段各类任务的负载分布与变化趋势。我们识别出序列长度分布不均导致的GPU空转、动态负载下并行策略低效、数据管理机制不佳及负载不均衡等问题。基于观察结果，本文提出PolyTrace基准套件，用于在真实负载下评估系统性能，并通过实际用例验证其准确率达到94.7%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are now widely used across many domains. With their rapid development, Reinforcement Learning with Verifiable Rewards (RLVR) has surged in recent months to enhance their reasoning and understanding abilities. However, its complex data flows and diverse tasks pose substantial challenges to RL training systems, and there is limited understanding of RLVR from a system perspective. To thoroughly understand the system challenges introduced by RLVR, we present a characterization study of RLVR tasks in our LLM deployment. Specifically, we investigate the distribution and variation trends of workloads across different RL tasks across training steps. We identify issues such as GPU idling caused by skewed sequence length distribution, inefficient parallel strategies in dynamically varying workloads, inefficient data management mechanisms, and load imbalance. We describe our observations and call for further investigation into the remaining open challenges. Furthermore, we propose PolyTrace benchmark suite to conduct evaluation with realistic workloads, and a practical use case validates that PolyTrace benchmark suite exhibits 94.7% accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。