通过自适应控制训练与推理差异，提升大模型强化学习的稳定性。
ACRL: Adaptive Control of Training-Inference Discrepancy for Stable Reinforcement Learning
- 动态调节训练与推理间的差异，避免因架构和精度不一致导致的不稳定。
- 在FP8量化下保持差异合理范围，训练稳定且性能媲美BF16基准。
- 提升策略熵促进探索，适合高精度需求的RL应用如语言模型优化。
大语言模型的强化学习训练常因训练与推理之间的差异而出现不稳定问题。这种差异主要源于训练与推理引擎的架构分离，以及推理采用低精度量化（如FP8）而训练使用更高精度计算。为缓解由高训练-推理差异引发的训练不稳定性，本文提出自适应控制强化学习（ACRL），通过自适应调控训练-推理差异，使其维持在合理范围内，从而确保训练稳定。除了提升稳定性，ACRL还自然增加策略熵，增强探索能力并提升准确性。实验表明，当推理端使用FP8量化时，ACRL能持续将训练-推理差异控制在合理区间，实现稳定训练。此外，其性能不仅达到BF16基线水平，还优于重要性采样（IS）修正方法。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) training for Large Language Models (LLMs) often suffers from instability due to the discrepancy between training and inference. This training-inference discrepancy stems from two primary factors: an architectural separation between training and inference engines, and the use of low-precision quantization in inference versus higher-precision computation in training. To address training instability issues caused by high training-inference discrepancy, we present the principles and methods for its adaptive control. We propose Adaptive Control Reinforcement Learning (ACRL), which adaptively maintains the training-inference discrepancy within a reasonable range to ensure stable RL training. Beyond stabilization, ACRL inherently increases policy entropy, thereby enhancing exploration and improving accuracy. The experimental results show that when the inference engine utilizes FP8 quantization, ACRL consistently maintains the training-inference discrepancy within a reasonable range and stabilizes RL training. Furthermore, ACRL not only matches the accuracy of the BF16 baseline but also outperforms importance sampling (IS) fixes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。