通过概率组合提升多模态模型融合效果,无需重新训练。
Unleashing the Potentials of Likelihood Composition for Multi-modal Language Models
- 用候选答案的对数概率做融合,设计去偏、突出、投票等操作
- 在9个VQA数据集上,混合组合方法优于简单集成和投票
- 适合想快速融合不同模型且不改架构的研究者
模型融合始终是重要课题,尤其在大语言模型(LLM)和多模态语言模型(MLM)架构、参数量与训练流程各异的背景下。本文提出一种后处理框架——似然组合(likelihood composition),旨在无需重训练即可融合异构模型。核心思想是在多选视觉问答任务中,组合多个模型的似然分布(即候选答案的对数概率)。我们引入去偏、突出、多数投票和集成等基本操作,通过组合形成混合组合方法(mix-composition)。在9个VQA数据集和10个MLM上的实验证明,该方法显著优于简单集成或多数投票。该框架支持新组合方式的拓展,期望为异构模型融合提供新视角并激发后续探索。
原文摘要 · Abstract (English)
Model fusing has always been an important topic, especially in an era where large language models (LLM) and multi-modal language models (MLM) with different architectures, parameter sizes and training pipelines, are being created all the time. In this work, we propose a post-hoc framework, aiming at fusing heterogeneous models off-the-shell, which we call \textit{likelihood composition}, and the basic idea is to compose multiple models' likelihood distribution when doing a multi-choice visual-question-answering task. Here the core concept, \textit{likelihood}, is actually the log-probability of the candidate answer. In \textit{likelihood composition}, we introduce some basic operations: \textit{debias}, \textit{highlight}, \textit{majority-vote} and \textit{ensemble}. By combining (composing) these basic elements, we get the mixed composition methods: \textit{mix-composition}. Through conducting comprehensive experiments on 9 VQA datasets and 10 MLMs, we prove the effectiveness of \textit{mix-composition} compared with simple \textit{ensemble} or \textit{majority-vote} methods. In this framework, people can propose new basic composition methods and combine them to get the new mixed composition methods. We hope our proposed \textit{likelihood composition} can provide a new perspective of fusing heterogeneous models and inspire the exploration under this framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。