用稀疏化提升多任务强化学习的适应能力,让模型更灵活。
Sparsity-Driven Plasticity in Multi-Task Reinforcement Learning
- 通过渐进式剪枝和稀疏进化训练增强模型可塑性
- 稀疏模型显著减少神经元休眠与表征坍塌现象
- 适合追求高效灵活多任务系统的研究者使用
可塑性衰退是深度强化学习中的关键挑战,尤其在多任务强化学习(MTRL)中,模型需应对多样且可能冲突的任务需求。本文系统研究了稀疏化方法(如渐进式剪枝GMP、稀疏进化训练SET)对MTRL中可塑性的提升作用。在共享主干、专家混合模型、正交专家混合等不同架构上,基于标准MTRL基准测试,对比密集基线及多种正则化方法。结果表明,GMP与SET能有效缓解神经元休眠与表征坍塌等可塑性退化指标。稀疏模型常优于密集模型,在多任务性能上达到与显式可塑性干预相当的水平。研究揭示了可塑性、网络稀疏性与MTRL设计间的复杂关系,证明动态稀疏化是一种鲁棒但依赖上下文的工具,有助于构建更具适应性的MTRL系统。
原文摘要 · Abstract (English)
Plasticity loss, a diminishing capacity to adapt as training progresses, is a critical challenge in deep reinforcement learning. We examine this issue in multi-task reinforcement learning (MTRL), where higher representational flexibility is crucial for managing diverse and potentially conflicting task demands. We systematically explore how sparsification methods, particularly Gradual Magnitude Pruning (GMP) and Sparse Evolutionary Training (SET), enhance plasticity and consequently improve performance in MTRL agents. We evaluate these approaches across distinct MTRL architectures (shared backbone, Mixture of Experts, Mixture of Orthogonal Experts) on standardized MTRL benchmarks, comparing against dense baselines, and a comprehensive range of alternative plasticity-inducing or regularization methods. Our results demonstrate that both GMP and SET effectively mitigate key indicators of plasticity degradation, such as neuron dormancy and representational collapse. These plasticity improvements often correlate with enhanced multi-task performance, with sparse agents frequently outperforming dense counterparts and achieving competitive results against explicit plasticity interventions. Our findings offer insights into the interplay between plasticity, network sparsity, and MTRL designs, highlighting dynamic sparsification as a robust but context-sensitive tool for developing more adaptable MTRL systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。