首次在强化学习中实现N:M结构化稀疏，兼顾性能与硬件加速。
RNM-TD3: N:M Semi-structured Sparse Reinforcement Learning From Scratch
- 在TD3框架中全程保持行级N:M结构稀疏训练
- 2:4稀疏下抗环境性能提升14%，87.5%稀疏仍具竞争力
- 适配支持N:M稀疏计算的硬件，利于实际部署
稀疏性是压缩深度神经网络的有效方法，可在不损失性能的前提下减少参数量。在深度强化学习中，神经网络仅保留原始权重的5%仍可实现接近稠密模型的性能。然而，现有方法多依赖非结构化的细粒度稀疏，导致计算模式不规则，限制了硬件加速。结构化的粗粒度稀疏虽支持硬件加速，但通常损害性能并增加剪枝复杂度。本文首次研究了强化学习中的N:M结构化稀疏，平衡压缩率、性能与硬件效率。所提RNM-TD3框架在离策略强化学习（TD3）中对所有网络强制执行行级N:M稀疏，兼容支持N:M稀疏矩阵运算的加速器。连续控制基准测试显示，该稀疏代理在50%-75%稀疏度（如2:4、1:4）下优于稠密基线，2:4稀疏时在Ant环境中性能最高提升14%；即使在87.5%稀疏（1:8）下仍保持竞争力，并具备潜在训练加速能力。
原文摘要 · Abstract (English)
Sparsity is a well-studied technique for compressing deep neural networks (DNNs) without compromising performance. In deep reinforcement learning (DRL), neural networks with up to 5% of their original weights can still be trained with minimal performance loss compared to their dense counterparts. However, most existing methods rely on unstructured fine-grained sparsity, which limits hardware acceleration opportunities due to irregular computation patterns. Structured coarse-grained sparsity enables hardware acceleration, yet typically degrades performance and increases pruning complexity. In this work, we present, to the best of our knowledge, the first study on N:M structured sparsity in RL, which balances compression, performance, and hardware efficiency. Our framework enforces row-wise N:M sparsity throughout training for all networks in off-policy RL (TD3), maintaining compatibility with accelerators that support N:M sparse matrix operations. Experiments on continuous-control benchmarks show that RNM-TD3, our N:M sparse agent, outperforms its dense counterpart at 50%-75% sparsity (e.g., 2:4 and 1:4), achieving up to a 14% increase in performance at 2:4 sparsity on the Ant environment. RNM-TD3 remains competitive even at 87.5% sparsity (1:8), while enabling potential training speedups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。