让大模型在保持效率的同时，精准识别不确定预测。
Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers
- 将贝叶斯方法聚焦于专家选择阶段，降低计算开销。
- 噪声下路由稳定性提升38%，校准误差降低94%。
- 适合需要可靠置信度的大模型应用开发者。
基础模型在需评估输出不确定性的场景中部署日益重要。尽管贝叶斯方法可提供严谨的不确定性量化，但其计算开销限制了其在大规模模型中的使用。当前顶尖模型通过精心设计的稀疏结构（如专家混合层，MoE）实现万亿级参数量。本文提出变分专家路由（VMoER），一种面向MoE层的结构化贝叶斯方法，用于建模不确定性。该方法将贝叶斯推断限制在专家选择阶段，通常由确定性路由网络完成。我们采用两种推断策略：对路由logits进行近似变分推断，以及推断用于随机专家选择的温度参数。在多个微调的基础模型上测试，VMoER在噪声环境下路由稳定性提升38%，校准误差减少94%，分布外检测的AUROC提高12%，且额外计算量不足1% FLOPs。结果表明，VMoER为构建稳健、具备不确定性感知能力的大模型提供了可扩展路径。
原文摘要 · Abstract (English)
Foundation models are increasingly being deployed in contexts where understanding the uncertainty of their outputs is critical to ensuring responsible deployment. While Bayesian methods offer a principled approach to uncertainty quantification, their computational overhead renders their use impractical for training or inference at foundation model scale. State-of-the-art models achieve parameter counts in the trillions through carefully engineered sparsity including Mixture-of-Experts (MoE) layers. In this work, we demonstrate calibrated uncertainty at scale by introducing Variational Mixture-of-Experts Routing (VMoER), a structured Bayesian approach for modelling uncertainty in MoE layers. VMoER confines Bayesian inference to the expert-selection stage which is typically done by a deterministic routing network. We instantiate VMoER using two inference strategies: amortised variational inference over routing logits and inferring a temperature parameter for stochastic expert selection. Across fine-tuning tested foundation models, VMoER improves routing stability under noise by 38\%, reduces calibration error by 94\%, and increases out-of-distribution AUROC by 12\%, while incurring less than 1\% additional FLOPs. These results suggest VMoER offers a scalable path toward robust and uncertainty-aware foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。