用信息论选大模型组合，比挑最准的还有效
Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection
- 基于互信息贪心选择模型，提升组合性能
- 实验证明相同预算下优于现有方法，尤其在高相关性场景
- 适合资源有限但需稳定高精度的落地应用
大语言模型常通过集成提升可靠性，但实际中模型间存在强相关性。本文将预算受限的集成选择问题建模为最大化真实标签与所选模型预测间的互信息。为解释为何模型增多性能仍会饱和，我们使用高斯-拷贝函数建模模型误差相关性，揭示了集成性能的信息论下限。基于此，提出一种直接从数据估计信息项的贪心算法，迭代构建满足查询预算的集成。在MEDMCQA、MMLU和IMDB电影评论三个问答与情感分类数据集上测试，结果表明，在相同查询预算下，本方法始终优于强基线。
原文摘要 · Abstract (English)
Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated. This raises a fundamental question: which models should be selected when forming an LLM ensemble? We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models. Furthermore, to explain why performance can saturate even with many models, we model the correlated errors of the models using Gaussian-copula and show an information-theoretic error floor for the performance of the ensemble. Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data and iteratively builds an ensemble under a query budget. We test our approach in two question answering datasets and one binary sentiment classification dataset: MEDMCQA, MMLU, and IMDB movie reviews. Across all datasets, we observe that our method consistently outperforms strong baselines under the same query budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。