用多个大模型的差异性提升预测可靠性，让不确定度更可信。
Simple Yet Effective: An Information-Theoretic Approach to Multi-LLM Uncertainty Quantification
- 通过信息论方法筛选并聚合表现一致的多模型子集
- 在二分类任务中校准度和预测性能优于单模型和简单集成
- 可作为提示词蒸馏的指导信号，优化模型校准能力
大语言模型在不同输入下常表现出不一致性，暗示其存在不确定性，尤其在高风险场景中亟需量化。以往校准与不确定性量化研究多聚焦于单个模型，忽视了模型多样性带来的潜力。我们假设：由于训练差异及语言的齐普夫分布特性，不同大模型会做出互补性预测，聚合其输出可获得更可靠的不确定性估计。为此，我们提出 MUSE（基于子集集成的多模型不确定性量化），一种简洁的信息论方法，利用詹森-香农散度识别并聚合校准良好的模型子集。在二分类任务上的实验表明，该方法在校准度和预测性能上均优于单模型及朴素集成基线。此外，我们探索将 MUSE 作为链式思维蒸馏的引导信号，用于微调模型以提升其校准能力。代码已开源：https://github.com/LARK-NLP-Lab/MUSE。
原文摘要 · Abstract (English)
Large language models (LLMs) often behave inconsistently across inputs, indicating uncertainty and motivating the need for its quantification in high-stakes settings. Prior work on calibration and uncertainty quantification often focuses on individual models, overlooking the potential of model diversity. We hypothesize that LLMs make complementary predictions due to differences in training and the Zipfian nature of language, and that aggregating their outputs leads to more reliable uncertainty estimates. To leverage this, we propose MUSE (Multi-LLM Uncertainty via Subset Ensembles), a simple information-theoretic method that uses Jensen-Shannon Divergence to identify and aggregate well-calibrated subsets of LLMs. Experiments on binary prediction tasks demonstrate improved calibration and predictive performance compared to single-model and naïve ensemble baselines. In addition, we explore using MUSE as guided signals with chain-of-thought distillation to fine-tune LLMs for calibration. MUSE is available at:https://github.com/LARK-NLP-Lab/MUSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。