提出自适应正则化方法,精准控制稀疏训练的稀疏率。
Adaptive Regularization for Sparsity Control in Bregman-Based Optimizers
- 根据当前稀疏度与目标稀疏度的差距动态调整正则化参数λ。
- 在75%至99%范围内稳定达到指定稀疏率,收敛更快且性能更优。
- 适用于需要高稀疏率的轻量化模型部署场景,尤其适合Bregman优化器。
稀疏训练可降低深度神经网络的内存和计算开销。然而,如添加ℓ₁惩罚的稀疏优化方法通常仅通过正则化参数λ间接控制稀疏度,而λ与最终稀疏率之间的映射关系不明确。实验发现,该敏感性在基于Bregman的优化器(如LinBreg和AdaBreg)中尤为显著:两者达到相同稀疏度时,λ值相差可达两个数量级,需耗费大量试错调参。为此,我们提出一种自适应正则化方案,依据当前模型稀疏度与目标稀疏度的差异动态更新λ。我们在ECAPA-TDNN和ResNet34上对自动说话人验证任务(VoxCeleb、CNCeleb)进行了评估。结果表明,该方法能可靠实现75%~99%的稀疏率目标;早期训练阶段收敛速度优于经过理想调参的非自适应基线,且最终等错误率(EER)匹配或超越基线。此外,该自适应方案继承了其非自适应版本的关键特性,包括对分布外数据的更强鲁棒性。
原文摘要 · Abstract (English)
Sparse training reduces the memory and computational costs of deep neural networks. However, sparse optimization methods, e.g., those adding an $\ell_1$ penalty, often control sparsity only indirectly through a regularization parameter $λ$, whose mapping to the final sparsity rate is non-trivial. In our experiments, we found this parameter sensitivity to be particularly pronounced for Bregman-based optimizers. Specifically, the two variants LinBreg and AdaBreg reach the same sparsity at $λ$ values that differ by up to two orders of magnitude, requiring expensive trial-and-error sweeps to achieve a user-specified sparsity. To address this, we propose an adaptive regularization scheme that updates $λ$ based on the difference between the model's current sparsity and the target sparsity. We analyze the resulting algorithm and evaluate it on automatic speaker verification with ECAPA-TDNN and ResNet34 on VoxCeleb and CNCeleb. The proposed method reliably achieves sparsity targets ranging between 75% and 99%. It also converges faster than the oracle-tuned non-adaptive baseline during early training and matches or surpasses its final performance in equal error rate. We further show that the adaptive scheme inherits key properties from its non-adaptive counterpart, including improved out-of-distribution robustness over the dense baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。