简单随机剪枝让深度强化学习模型更易扩展，效果优于密集网络。
Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning
- 训练前一次性随机剪枝，仅移除固定比例权重。
- 稀疏网络在参数效率和抗优化问题上显著优于密集网络。
- 适合追求模型可扩展性的强化学习研究者使用。
由于训练过程中的网络病态问题，深度强化学习模型的有效扩展一直非常困难，促使人们提出各种针对性干预措施，如周期性重置和层归一化等架构改进。我们发现，仅通过引入静态网络稀疏性，即可在现有先进架构下实现超越密集网络的进一步扩展潜力。该方法采用简单的单次随机剪枝：在训练前一次性随机移除预设比例的网络权重。分析表明，与直接扩展密集网络相比，此类稀疏网络在参数效率和对塑性损失、梯度干扰等优化挑战的抵抗能力方面均有显著提升。我们还将评估扩展至视觉和流式强化学习场景，结果一致显示网络稀疏性的优势。
原文摘要 · Abstract (English)
Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motivating various targeted interventions such as periodic reset and architectural advances such as layer normalization. Instead of pursuing more complex modifications, we show that introducing static network sparsity alone can unlock further scaling potential beyond their dense counterparts with state-of-the-art architectures. This is achieved through simple one-shot random pruning, where a predetermined percentage of network weights are randomly removed once before training. Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity and stronger resistance to optimization challenges like plasticity loss and gradient interference. We further extend our evaluation to visual and streaming RL scenarios, demonstrating the consistent benefits of network sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。