让专家门控随置信度自适应调整，提升MoE模型性能。
Confidence-Adaptive SwiGLU for Mixture-of-Experts
- 根据令牌路由置信度动态调节门控陡峭程度
- 在8-28层模型上提升平均CORE指标表现
- 参数增量极小，计算开销可控，适合实际部署
SwiGLU已成为现代Transformer MLP的标准门控激活函数，但其门控陡峭度（即门控函数的平滑性与选择性）通常在训练中固定不变。本文提出一种针对混合专家（MoE）模型的改进版SwiGLU——$κ$-SwiGLU，可根据令牌级路由置信度动态调整专家门控的陡峭程度。具体而言，$κ$-SwiGLU将SiLU门控陡峭系数参数化为路由器logit的可学习函数，使每个门控单元能在平滑广覆盖和尖锐选择性之间进行插值。我们在涵盖8至28层的MoE Transformer模型上，于FineWeb-Edu数据集上评估了$κ$-SwiGLU。结果表明，该方法在几乎不增加参数量且仅带来轻微计算开销的前提下，显著提升了平均CORE性能，验证了基于置信度的门控自适应机制在优化MoE MLP中的有效性。代码已开源：https://github.com/askerlee/kappa-swiglu。
原文摘要 · Abstract (English)
SwiGLU has become a standard gated activation in modern Transformer MLPs, yet its gate sharpness -- the smoothness and selectivity of the gating function -- is typically fixed throughout training. In this work, we propose Confidence-Aware SwiGLU ($κ$-SwiGLU), a variant of SwiGLU for Mixture-of-Experts (MoE) models that adjusts expert gate sharpness according to token-level routing confidence. Specifically, $κ$-SwiGLU parameterizes the SiLU gate sharpness coefficient as a learnable function of the router logit, enabling each expert gate unit to interpolate between smooth, broadly active gating and sharp, selective gating. We evaluate $κ$-SwiGLU on the FineWeb-Edu dataset across MoE Transformer models ranging from 8 to 28 layers. Across these settings, $κ$-SwiGLU improves mean CORE performance while adding negligible parameters and incurring only a small computational overhead, demonstrating that confidence-aware gate sharpness is a promising mechanism for improving MoE MLPs. The code is available at https://github.com/askerlee/kappa-swiglu.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。