arXiv:2505.11416cs.NEcs.AI2025-05被引 2

动态选择关键神经元,让模型计算更省力还更准。

MID-L: Matrix-Interpolated Dropout Layer with Layer-wise Neuron Selection

  • 用可学习的门控向量动态选神经元,输入不同时激活不同路径。
  • 平均减少55%活跃神经元,计算量降低1.7倍,精度不降反升。
  • 适合想提速、省资源或研究高效推理的开发者和研究者。

现代神经网络通常对每个输入激活全部神经元,造成不必要的计算开销。本文提出矩阵插值丢弃层(MID-L),通过学习的、依赖输入的门控向量,在两条变换路径间进行插值,动态选择并激活最具信息量的神经元。与传统丢弃或静态稀疏方法不同,MID-L采用可微的Top-k掩码策略,实现每输入自适应计算且保持端到端可训练性。该模块与模型无关,可无缝集成至现有架构中。在六个基准数据集(包括MNIST、CIFAR-10、CIFAR-100、SVHN、UCI Adult和IMDB)上的大量实验表明,MID-L可实现平均55%的活跃神经元减少、1.7倍的浮点运算节省,同时保持或超越基线精度。通过切片互信息(SMI)验证了所选神经元的信息量与选择性,并观察到在过拟合和噪声数据下具有更强鲁棒性。此外,MID-L表现出优异的推理延迟和内存占用特性,适用于科研探索及计算资源受限场景。这些结果表明,MID-L是一种通用、即插即用的动态计算模块,弥合了丢弃正则化与高效推理之间的差距。

原文摘要 · Abstract (English)

Modern neural networks often activate all neurons for every input, leading to unnecessary computation and inefficiency. We introduce Matrix-Interpolated Dropout Layer (MID-L), a novel module that dynamically selects and activates only the most informative neurons by interpolating between two transformation paths via a learned, input-dependent gating vector. Unlike conventional dropout or static sparsity methods, MID-L employs a differentiable Top-k masking strategy, enabling per-input adaptive computation while maintaining end-to-end differentiability. MID-L is model-agnostic and integrates seamlessly into existing architectures. Extensive experiments on six benchmarks, including MNIST, CIFAR-10, CIFAR-100, SVHN, UCI Adult, and IMDB, show that MID-L achieves up to average 55\% reduction in active neurons, 1.7$\times$ FLOPs savings, and maintains or exceeds baseline accuracy. We further validate the informativeness and selectivity of the learned neurons via Sliced Mutual Information (SMI) and observe improved robustness under overfitting and noisy data conditions. Additionally, MID-L demonstrates favorable inference latency and memory usage profiles, making it suitable for both research exploration and deployment on compute-constrained systems. These results position MID-L as a general-purpose, plug-and-play dynamic computation layer, bridging the gap between dropout regularization and efficient inference.

动态计算神经元剪枝效率优化可微掩码

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。