让大模型强化学习在低内存下稳定运行，突破长序列训练瓶颈。
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
- 通过稀疏采样与偏差校正实现低内存强化学习
- 内存开销降低同时保持性能，支持稀疏推理部署
- 适合资源受限场景下的大模型训练与推理
强化学习（RL）已成为激发大语言模型（LLM）复杂推理能力的关键方法。然而，长时间序列回放过程中存储键值（KV）缓存带来的巨大内存开销构成关键瓶颈，常导致在有限硬件上难以高效训练。现有KV压缩技术虽可缓解推理压力，但直接用于RL训练会引发严重策略不匹配，导致性能崩溃。为此，本文提出Sparse-RL，实现稀疏回放下的稳定强化学习训练。我们发现不稳定源于密集旧策略、稀疏采样策略与学习者策略之间的根本性策略不匹配。为解决该问题，Sparse-RL引入感知稀疏性的拒绝采样与基于重要性的重加权机制，以纠正压缩带来的信息损失所引起的离策略偏差。实验表明，Sparse-RL在显著降低回放开销的同时保持了与密集基线相当的性能。此外，该方法天然支持稀疏感知训练，显著提升了模型在稀疏推理部署中的鲁棒性。相关训练数据与代码已公开于仓库。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has become essential for eliciting complex reasoning capabilities in Large Language Models (LLMs). However, the substantial memory overhead of storing Key-Value (KV) caches during long-horizon rollouts acts as a critical bottleneck, often prohibiting efficient training on limited hardware. While existing KV compression techniques offer a remedy for inference, directly applying them to RL training induces a severe policy mismatch, leading to catastrophic performance collapse. To address this, we introduce Sparse-RL empowers stable RL training under sparse rollouts. We show that instability arises from a fundamental policy mismatch among the dense old policy, the sparse sampler policy, and the learner policy. To mitigate this issue, Sparse-RL incorporates Sparsity-Aware Rejection Sampling and Importance-based Reweighting to correct the off-policy bias introduced by compression-induced information loss. Experimental results show that Sparse-RL reduces rollout overhead compared to dense baselines while preserving the performance. Furthermore, Sparse-RL inherently implements sparsity-aware training, significantly enhancing model robustness during sparse inference deployment. The corresponding training data and code are publicly available on the repository.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。