用量化提升大模型强化学习效率，加速训练并增强探索能力。
QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
- 结合NVFP4量化与LoRA，加速强化学习推理阶段。
- 单张H100 GPU训练320亿参数模型，速度提升1.5倍以上。
- 动态噪声机制促进探索，7B模型达全量微调性能水平。
我们提出QeRL，一种面向大语言模型（LLM）的量化增强强化学习框架。尽管强化学习对提升模型推理能力至关重要，但其资源消耗巨大，需大量GPU内存和长时间推理周期。QeRL通过将NVFP4量化与低秩适配（LoRA）结合，在加速强化学习推理阶段的同时降低内存开销。除效率提升外，研究发现量化噪声可增加策略熵，增强探索能力，有助于在强化学习中发现更优策略。为此，QeRL引入自适应量化噪声（AQN）机制，动态调节训练过程中的噪声强度。实验表明，QeRL在推理阶段实现超过1.5倍的速度提升。这是首个可在单张H100 80GB GPU上训练320亿参数模型的强化学习框架，整体训练效率显著提升。相比16位LoRA和QLoRA，QeRL具有更快的奖励增长速度和更高最终准确率；在数学基准测试（如GSM8K 90.8%、MATH 500 77.4%）中，7B模型表现与全参数微调相当。这些结果确立了QeRL在大模型强化学习训练中的高效性与有效性。
原文摘要 · Abstract (English)
We propose QeRL, a Quantization-enhanced Reinforcement Learning framework for large language models (LLMs). While RL is essential for LLMs' reasoning capabilities, it is resource-intensive, requiring substantial GPU memory and long rollout durations. QeRL addresses these issues by combining NVFP4 quantization with Low-Rank Adaptation (LoRA), accelerating rollout phase of RL while reducing memory overhead. Beyond efficiency, our findings show that quantization noise increases policy entropy, enhancing exploration, and enabling the discovery of better strategies during RL. To further optimize exploration, QeRL introduces an Adaptive Quantization Noise (AQN) mechanism, which dynamically adjusts noise during training. Experiments demonstrate that QeRL delivers over 1.5 times speedup in the rollout phase. Moreover, this is the first framework to enable RL training of a 32B LLM on a single H100 80GB GPU, while delivering overall speedups for RL training. It also achieves faster reward growth and higher final accuracy than 16-bit LoRA and QLoRA, while matching the performance of full-parameter fine-tuning on mathematical benchmarks such as GSM8K (90.8%) and MATH 500 (77.4%) in the 7B model. These results establish QeRL as an efficient and effective framework for RL training in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。