用量化模型加速大模型强化学习的推理过程，提升训练效率。
QuRL: Efficient Reinforcement Learning with Quantized Rollout
- 用量化智能体替代全精度模型进行推理，显著提速
- 在两个数据集上实现20%至80%的推理速度提升
- 适合需要高效训练推理类大模型的研究者
基于可验证奖励的强化学习（RLVR）已成为训练推理型大语言模型的主流范式。然而，由于大模型的自回归解码特性，推理过程成为强化学习训练的效率瓶颈，占总训练时间的高达70%。本文提出量化强化学习（QuRL），采用量化智能体加速推理过程。针对两大挑战：一是提出自适应裁剪范围（ACR），根据全精度与量化智能体间的策略比例动态调整裁剪比率，有效缓解长期训练崩溃问题；二是发现权重更新过小导致量化难以捕捉变化的问题，通过不变缩放技术降低量化噪声并增强权重更新。我们在DeepScaleR和DAPO数据集上分别使用INT8和FP8量化进行实验，实现了训练期间20%至80%的推理加速。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70\% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。