用几何方法精准量化专家模型的分工程度，还能提前发现训练失败。
Geometric Metrics for MoE Specialization: From Fisher Information to Early Failure Detection
- 基于费雪信息度量构建专家路由的几何分析框架。
- 新指标与下游性能相关性达0.91，训练10%时可预测失败（AUC=0.89）。
- 适合关注模型可解释性与训练稳定性的研究者和工程师。
专家分工是混合专家（MoE）模型成功的关键，但现有度量（余弦相似度、路由熵）缺乏理论基础，且在重参数化下结果不一致。本文提出首个基于信息几何的严谨分析框架，揭示专家路由分布演化于概率单纯形上，由费雪信息度量定义。证明传统启发式度量违反参数化不变性（定理1），确立分工对应测地线流并给出近似界（定理2），推导出具有理论阈值的失败预测器（定理3）。提出两个新指标：费雪专业化指数（FSI）与下游性能相关性r=0.91±0.02；费雪异质性得分（FHS）在训练完成10%时即可预测失败，AUC=0.89±0.03，优于基于验证损失的早停策略23%，且计算开销低40倍。实验涵盖语言建模（WikiText-103, C4）、视觉MoE（ImageNet）及规模研究（8-64专家，125M-2.7B参数），验证理论预测。干预协议在检测到FHS>1时可实现87%恢复率。
原文摘要 · Abstract (English)
Expert specialization is fundamental to Mixture-of-Experts (MoE) model success, yet existing metrics (cosine similarity, routing entropy) lack theoretical grounding and yield inconsistent conclusions under reparameterization. We present an information-geometric framework providing the first rigorous characterization of MoE specialization dynamics. Our key insight is that expert routing distributions evolve on the probability simplex equipped with the Fisher information metric, enabling formal analysis via Riemannian geometry. We prove that standard heuristic metrics violate parameterization invariance (Theorem 1), establish that specialization corresponds to geodesic flow with quantified approximation bounds (Theorem 2), and derive a failure predictor with theoretical threshold justification (Theorem 3). The framework introduces two principled metrics: Fisher Specialization Index (FSI) achieving r=0.91+/-0.02 correlation with downstream performance, and Fisher Heterogeneity Score (FHS) predicting training failure at 10% completion with AUC=0.89+/-0.03 -- outperforming validation-loss-based early stopping by 23% while requiring 40x fewer compute cycles. We validate intervention protocols achieving 87% recovery rate when FHS>1 is detected. Comprehensive experiments across language modeling (WikiText-103, C4), vision MoE (ImageNet), and scaling studies (8-64 experts, 125M-2.7B parameters) validate our theoretical predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。