orthogonality正则化无法提升MoE专家多样性，反而加剧权重重叠。
Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations
- 用正交性约束强制专家差异，但效果适得其反
- 权重重叠最高增加114%，激活重叠仍达0.6
- 性能波动大，不适合用于提升MoE多样性
Mixture-of-Experts (MoE) 模型通过稀疏激活实现高效计算，但几何正则化在促进专家专一化中的作用尚不明确。我们引入正交性损失以增强专家多样性，却发现其在多个方面失效：不仅未降低权重空间重叠（MSO反而上升最多114%），激活空间重叠仍维持在约0.6的高水平；且对性能影响不一致——在WikiText-103上仅微升-0.9%，在TinyStories上轻微下降+0.9%，在PTB上结果标准差超过1.0。7种正则强度下的分析显示，权重与激活正交性间无显著相关性（r = -0.293, p = 0.523）。结果表明，权重空间正则化既未达成几何目标，也无法稳定提升性能，因此不适用于MoE多样性建设。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (~0.6) regardless of regularization, and effects on performance are inconsistent -- marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。