为微调的大模型提供无需修改训练的不确定性估计方法
Bayesian Mixture of Experts For Large Language Models
- 基于专家混合架构，对每个专家的第二层应用结构化拉普拉斯近似
- 在常见推理基准上，显著降低期望校准误差和负对数似然
- 适合需要可靠置信度判断的下游任务，如医疗或金融决策
我们提出贝叶斯专家混合（Bayesian-MoE），一种针对微调大语言模型的后处理不确定性估计框架，基于专家混合架构。该方法对每个专家的第二线性层应用结构化拉普拉斯近似，实现校准的不确定性估计，无需修改原始训练流程或引入新参数。与以往在附加适配器模块上进行贝叶斯推断的方法不同，Bayesian-MoE 直接作用于专家路径中已存在的组件，利用其模块化设计实现可计算的分块后验估计。我们采用克罗内克分解的低秩近似来建模曲率，并推导出可扩展的预测不确定性与边缘似然估计。在 Qwen1.5-MoE 与 DeepSeek-MoE 的常识推理基准上的实验表明，Bayesian-MoE 在预期校准误差（ECE）和负对数似然（NLL）上均优于基线，验证了其在可靠下游决策中的有效性。
原文摘要 · Abstract (English)
We present Bayesian Mixture of Experts (Bayesian-MoE), a post-hoc uncertainty estimation framework for fine-tuned large language models (LLMs) based on Mixture-of-Experts architectures. Our method applies a structured Laplace approximation to the second linear layer of each expert, enabling calibrated uncertainty estimation without modifying the original training procedure or introducing new parameters. Unlike prior approaches, which apply Bayesian inference to added adapter modules, Bayesian-MoE directly targets the expert pathways already present in MoE models, leveraging their modular design for tractable block-wise posterior estimation. We use Kronecker-factored low-rank approximations to model curvature and derive scalable estimates of predictive uncertainty and marginal likelihood. Experiments on common-sense reasoning benchmarks with Qwen1.5-MoE and DeepSeek-MoE demonstrate that Bayesian-MoE improves both expected calibration error (ECE) and negative log-likelihood (NLL) over baselines, confirming its effectiveness for reliable downstream decision-making.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。