不学路由的模块化专家模型，靠标签直接分配任务，效果更好。
SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specialization
- 用固定规则按类别分配专家，无需学习参数和额外损失。
- 在CIFAR-100和ImageNet上分别达到79.23%和79.89%准确率，超越基线。
- 适合需要高效模块化设计、追求轻量部署的研究者或工程师。
混合专家（MoE）网络通过学习路由机制实现专业化，但在相同参数预算下，学习路由的表现常不及不使用路由的基线模型。我们提出SpecDrop，一种无参数的固定路由方案：每个专家接收其对应类别的权重 $p_a$，并有微小泄漏权重 $p_i > 0$ 分配给其他类别，通过固定的分母合并输出，全程无学习参数与辅助损失，仅需推理时提供类别标签。在图像任务中（如在ResNet-110上的CIFAR-100，ViT-S/16上的ImageNet-1K），SpecDrop分别取得79.23%和79.89%的准确率，显著优于参数匹配的无路由基线（在CIFAR-100上+4.75，在ImageNet-1K上+6.53）。这些提升表明类别监督在路由中的实际价值——但若直接对密集模型掩码输出，精度更高（85.2 / 83.7）。SpecDrop的核心贡献在于将标签转化为训练中内化的模块结构：分支-类别对齐率达58%/100%，且掩码增益为0.00（CIFAR）/+1.06（ImageNet）。在模糊划分场景（如30M Transformer的SlimPajama-6B语言建模，或基于LoRA的Llama-3.2-1B指令调优），路由机制退化为等效于无路由控制，与理论预测一致。关键发现是：路由有效性取决于训练信号粒度与类别目标的对齐，而非路由算法本身。
原文摘要 · Abstract (English)
Mixture-of-experts (MoE) networks pursue specialization through learned routers, gates, and load-balancing losses, yet at matched total-parameter budgets learned routers can underperform equal-weight No-Routing baselines. Is the bottleneck the routing algorithm, or the alignment between training-signal granularity and the target categories? We probe the question with SpecDrop, a fixed parameter-free routing scheme: each of $K$ branches receives weight $p_a$ for its assigned category and a small leakage $p_i > 0$ otherwise, merged through a category-independent fixed denominator, with no learned routing parameters and no auxiliary losses; the category label is required at inference. On vision tasks where each image has one superclass label (CIFAR-100 on ResNet-110; ImageNet-1K on ViT-S/16), SpecDrop reaches 79.23% on CIFAR-100 and 79.89% on ImageNet-1K, exceeding parameter-matched baselines that do not use the label (+4.75 over dense on CIFAR-100; +6.53 over the No-Routing+SE control on ImageNet-1K). These gains quantify what category supervision buys when deployed through routing -- not an advantage over label-aware deployments of the baselines: given the same label, masking a dense model's outputs is stronger for accuracy alone (85.2 / 83.7). SpecDrop's contribution is converting the label into trained-in modular structure: 58%/100% branch-category alignment, and masking gains of 0.00 (CIFAR) / +1.06 (ImageNet) -- the output-space restriction is largely internalized during training. On fuzzy partitions, where training units span multiple categories (SlimPajama-6B language modeling with a 30M Transformer; SuperNI instruction tuning over Llama-3.2-1B with LoRA), the routing mechanism reduces to the matched No-Routing controls within seed noise, the null our thesis predicts. Granularity alignment, not algorithm choice, localizes when routing helps. Code: https://github.com/Beryex/SpecDrop
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。