稀疏MoE在视觉任务中何时有效？关键看计算资源利用率。
When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
- 通过控制路由策略与计算占比，发现高计算利用率是提升性能的前提。
- ImageNet上需双专家路由（k≥2）才能实现准确率提升，单一专家无效。
- 批处理维度的专家分配是导致模型失效的主要原因，局部路由更优。
混合专家（MoE）网络承诺在准确率与计算成本间取得良好平衡，但实际视觉部署受限于专家崩溃和端到端效率提升有限。本文研究稀疏top-k路由在视觉分类中的有效性，基于多种子协议在四个基准（CIFAR-10/100、Tiny-ImageNet、ImageNet-1K）上评估。发现存在一种‘计算利用率模式’：正向准确率差距要求总计算量的较大比例ρ被路由；在ImageNet规模下，这虽必要但不充分，还需多专家路由（k≥2）。两个受控实验验证该结论：在CIFAR-10上改变隐藏层大小，标准与深度卷积骨干网络均出现预测符号反转，排除骨干类型为关键变量；在ImageNet-1K上固定架构、初始化与ρ，仅调整top-k，五个种子中准确率差距由正转负。对Soft MoE的样本级变体（对专家而非批次进行softmax）在CIFAR-100上超越密集基线，揭示批处理维度分配是CNN局部设置下的主要失败原因。代码与汇总结果见：https://github.com/libophd/sparse-moe-vision-rho。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) networks promise favorable accuracy-compute trade-offs, yet practical vision deployments are hindered by expert collapse and limited end-to-end efficiency gains. We study when sparse top-$k$ routing with hard capacity constraints helps in vision classification, evaluated under multi-seed protocols on four benchmarks (CIFAR-10/100, Tiny-ImageNet, ImageNet-1K). We observe a \emph{compute-leverage pattern}: positive accuracy gaps require a substantial fraction $ρ$ of total FLOPs to be routed; at ImageNet scale this is necessary but not sufficient, as multi-expert routing ($k \geq 2$) is additionally required. Two controlled experiments isolate these factors. A hidden-size sweep on CIFAR-10 yields both predicted sign reversals across standard and depthwise backbones, ruling out backbone family as the active variable. An ImageNet-1K ablation that varies only top-$k$ -- holding architecture, initialization, and $ρ$ fixed -- reverses the gap from positive to negative across all five seeds. A per-sample variant of Soft MoE that softmaxes over experts rather than the batch rescues CIFAR-100 above the dense baseline, identifying batch-axis dispatch as the dominant failure mode in per-sample CNN settings. Code and aggregate results: https://github.com/libophd/sparse-moe-vision-rho.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。