让强化学习模型自动调整稀疏程度，兼顾高效与高性能。
Eau De $Q$-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learning
- 多版本网络并行训练，动态选择最优稀疏度
- 在Atari和MuJoCo上实现高稀疏度仍保持高精度
- 无需人工调参，适应学习进度自动优化
近期研究证明，稀疏的深度强化学习代理可与密集模型竞争，为推理时间与内存受限的应用带来可能。然而，现有从密集到稀疏的方法依赖人工设计的稀疏调度，且最终稀疏度作为超参数需反复调试，过高可能导致性能下降。本文提出Eau De $Q$-Network（EauDeQN），通过维护多个不同稀疏度的在线网络，共享一个目标网络进行训练。每次目标网络更新时，选择损失最小的在线网络作为新目标，并将其余网络修剪为该网络的稀疏版本。在Atari $2600$基准和MuJoCo物理模拟器上的实验表明，EauDeQN可在保持高性能的同时达到高稀疏度。
原文摘要 · Abstract (English)
Recent works have successfully demonstrated that sparse deep reinforcement learning agents can be competitive against their dense counterparts. This opens up opportunities for reinforcement learning applications in fields where inference time and memory requirements are cost-sensitive or limited by hardware. Until now, dense-to-sparse methods have relied on hand-designed sparsity schedules that are not synchronized with the agent's learning pace. Crucially, the final sparsity level is chosen as a hyperparameter, which requires careful tuning as setting it too high might lead to poor performances. In this work, we address these shortcomings by crafting a dense-to-sparse algorithm that we name Eau De $Q$-Network (EauDeQN). To increase sparsity at the agent's learning pace, we consider multiple online networks with different sparsity levels, where each online network is trained from a shared target network. At each target update, the online network with the smallest loss is chosen as the next target network, while the other networks are replaced by a pruned version of the chosen network. We evaluate the proposed approach on the Atari $2600$ benchmark and the MuJoCo physics simulator, showing that EauDeQN reaches high sparsity levels while keeping performances high.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。