arXiv:2603.16367cs.LGcs.AI2026-03

让MLP按输入动态计算,省算力还防过拟合。

DynamicGate MLP Conditional Computation via Learned Structural Dropout and Input Dependent Gating for Functional Plasticity

  • 用可学习的门控机制决定每层单元是否激活,实现输入相关计算。
  • 在多个数据集上比基线节省30%~60%计算量,精度不降。
  • 适合资源受限场景,如移动端或大规模模型推理。

Dropout是一种通过随机丢弃隐藏单元来缓解过拟合的正则化技术,而标准推理采用全网络密集计算,与条件计算(根据输入决定执行路径)的目标和机制不同。本文提出DynamicGate-MLP,将两者统一为一个框架:不使用随机掩码,而是学习门控机制决定每个单元(或模块)是否启用,从而在抑制冗余计算的同时实现样本依赖的执行,使计算聚焦于输入所需部分。为此,定义连续门控概率,在推理时生成离散执行掩码以选择路径;训练中通过惩罚期望门控使用率控制计算预算,并利用直通估计器(STE)优化离散掩码。在MNIST、CIFAR-10、Tiny-ImageNet、Speech Commands和PBMC3k上评估,与多种MLP基线及MoE变体对比,采用门控激活率和分层加权相对MAC数作为一致衡量标准,而非依赖硬件的运行时间。

原文摘要 · Abstract (English)

Dropout is a representative regularization technique that stochastically deactivates hidden units during training to mitigate overfitting. In contrast, standard inference executes the full network with dense computation, so its goal and mechanism differ from conditional computation, where the executed operations depend on the input. This paper organizes DynamicGate-MLP into a single framework that simultaneously satisfies both the regularization view and the conditional-computation view. Instead of a random mask, the proposed model learns gates that decide whether to use each unit (or block), suppressing unnecessary computation while implementing sample-dependent execution that concentrates computation on the parts needed for each input. To this end, we define continuous gate probabilities and, at inference time, generate a discrete execution mask from them to select an execution path. Training controls the compute budget via a penalty on expected gate usage and uses a Straight-Through Estimator (STE) to optimize the discrete mask. We evaluate DynamicGate-MLP on MNIST, CIFAR-10, Tiny-ImageNet, Speech Commands, and PBMC3k, and compare it with various MLP baselines and MoE-style variants. Compute efficiency is compared under a consistent criterion using gate activation ratios and a layerweighted relative MAC metric, rather than wall-clock latency that depends on hardware and backend kernels.

MLP动态计算门控机制高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。