让量化推理与训练对齐，加速大模型强化学习并保持稳定
QaRL: Rollout-Aligned Quantization-Aware RL for Fast and Stable Training under Training--Inference Mismatch
- 用量化感知的前向传播对齐推理端，减少训练-推理差异
- 在数学任务上比传统量化方法提升5.5分，且避免重复错误输出
- 适合追求高效稳定训练的大模型强化学习研究者
大语言模型强化学习常因生成回溯（rollout）耗时而变慢。近期方法通过量化加速解码，但导致训练与推理精度不一致，引发优化不稳定。本文提出QaRL（Rollout-Aligned Quantization-Aware RL），使训练侧前向传播与量化回溯对齐，缩小差距。进一步发现量化回溯中长文本易产生重复、混乱的错误令牌，为此引入TBPO（Trust-Band Policy Optimization），采用双截断机制处理负样本，确保更新在信任区域内。在Qwen3-30B-A3B MoE模型上解决数学问题时，QaRL相比传统量化方法提升5.5分，同时增强稳定性并保留低比特吞吐优势。
原文摘要 · Abstract (English)
Large language model (LLM) reinforcement learning (RL) pipelines are often bottlenecked by rollout generation, making end-to-end training slow. Recent work mitigates this by running rollouts with quantization to accelerate decoding, which is the most expensive stage of the RL loop. However, these setups destabilize optimization by amplifying the training-inference gap: rollouts are operated at low precision, while learning updates are computed at full precision. To address this challenge, we propose QaRL (Rollout Alignment Quantization-Aware RL), which aligns training-side forward with the quantized rollout to minimize mismatch. We further identify a failure mode in quantized rollouts: long-form responses tend to produce repetitive, garbled tokens (error tokens). To mitigate these problems, we introduce TBPO (Trust-Band Policy Optimization), a sequence-level objective with dual clipping for negative samples, aimed at keeping updates within the trust region. On Qwen3-30B-A3B MoE for math problems, QaRL outperforms quantized-rollout training by +5.5 while improving stability and preserving low-bit throughput benefits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。