用老虎机算法自动删神经元,让模型更轻更快还保持精度。
Structured Neuron Pruning in Deep Neural Networks Using Multi-Armed Bandits
- 把每个神经元当老虎机杠杆,试删后看损失变化来决定是否真删。
- 在分类、回归和深度任务上,UCB1等算法比传统剪枝方法更好。
- 适合想高效压缩模型又不想改代码的工程师和研究人员。
深度神经网络通常包含冗余隐藏单元。移除单个权重可减少参数量,但非结构化稀疏在标准密集实现中难以利用。本文提出一种基于多臂老虎机(MAB)的结构化剪枝框架,通过将候选神经元视为“杠杆”,临时屏蔽某神经元,测量小批量损失变化,再恢复并更新其安全移除收益估计。该框架支持随机策略(如Epsilon-Greedy、Softmax、UCB1、Thompson Sampling)和乘法权重策略(如Hedge风格与EXP3)。在表格分类、回归及涵盖图像、文本和推理任务的深度学习基准上评估,统计检验显示各方法存在显著差异。在表格分类任务中,UCB1获得最高平均排名,优于未剪枝模型;在回归任务中,UCB1表现最优,与多种标准回归模型相当或更优(按R²衡量);在深度学习任务中,UCB1与Thompson Sampling排名最强,多种MAB策略显著优于未剪枝模型、基于幅值剪枝和贪心激活变化剪枝。结果表明,基于MAB的神经元剪枝是有效且计算可行的结构化模型压缩方法。
原文摘要 · Abstract (English)
Deep neural networks often contain redundant hidden units. Removing individual weights can reduce parameter count, but unstructured sparsity is not always easy to exploit in standard dense implementations. This paper develops a structured pruning framework in which complete neurons are removed using multi-armed bandit (MAB) algorithms. Each candidate neuron is treated as an arm; pulling an arm temporarily masks that neuron, measures the change in loss on a sampled mini-batch, restores the neuron, and updates an estimate of its safe-removal reward. The framework supports stochastic policies, including Epsilon-Greedy, Softmax, UCB1 and Thompson Sampling, and multiplicative-weight policies, including Hedge-style multiplicative weights and EXP3. We evaluate the method on tabular classification, tabular regression and deep neural-network benchmarks covering image, text and reasoning tasks. Statistical comparisons using the Friedman test followed by the Nemenyi post-hoc test show significant differences between methods. On tabular classification tasks, UCB1 obtains the highest mean rank among pruning policies and improves on the unpruned neural network. On regression tasks, UCB1 obtains the highest mean rank and is statistically competitive with, or superior to, several standard regression models according to R^2. On deep-learning tasks, UCB1 and Thompson Sampling obtain the strongest ranks, and several MAB policies significantly outperform the unpruned model, magnitude-based neuron pruning and greedy activation-variation pruning. The results show that MAB-based neuron pruning is an effective and computationally practical approach for structured model reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。