提出通道专家混合模型,用稀疏通道选择提升卷积网络效率。
Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

- 将专家机制从卷积操作转向通道选择,每专家仅激活少数输入通道。
- 在ImageNet和CIFAR-100上减少16.7%计算量,延迟降低且性能不降。
- 适合追求高效推理的视觉模型部署,尤其适合资源受限场景。
Mixture-of-Experts(MoE)通过将每个输入路由到少量独立参数化的专家来扩展语言模型。我们发现,将此设计直接应用于卷积网络会因结构问题失败:并行卷积专家读取相同的输入通道时,学习到几乎相同的滤波器。因此,我们将专家维度从操作复制转向通道选择。提出结构化稀疏的通道混合层——通道专家混合(MoCE),受MoE启发,替代原有的点对点(1×1)通道压缩投影。在MoCE中,每个专家是一个输出通道,其具有由k << C个输入通道构成的稀疏支持。选定通道通过温度由输入预测的softmax组合,使专家可动态切换均值或最大值聚合模式。一个残差专家汇总未选通道信息,负载均衡损失确保通道覆盖完整。与代价随C²增长的密集投影相比,该机制相对开销仅随k/C变化,实测时间也验证了预期节省。在ResNet骨干、ImageNet-1K与CIFAR-100上的迁移学习、EfficientViT及现代训练方案中,MoCE在保持或超越基线性能的同时,将MACs降低16.7%,端到端延迟减少。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k << C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose relative cost scales as k/C, and the predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。