arXiv:2604.22823cs.CVcs.AI2026-04被引 3

提出PivotMerge框架,实现多模态预训练模型的对齐融合。

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

论文配图:PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging
图 1 · 摘自论文原文
  • 通过共享空间分解与筛选,分离跨域共性与干扰信息。
  • 基于对齐贡献动态分配层权重,解决不同层作用不均问题。
  • 在多个基准上超越现有方法,适合多源异构模型整合场景。

多模态大语言模型依赖于多样化数据源的多模态预训练,不同数据集常带来互补的跨模态对齐能力。模型融合为将多个具备互补优势的专家模型高效整合为统一模型提供了低成本方案。然而,现有研究主要聚焦后微调场景,预训练阶段的融合仍被忽视。我们认为,多模态预训练的核心在于建立有效的跨模态对齐,将视觉与文本表示映射至统一语义空间。受此启发,我们提出后对齐融合任务,旨在整合来自异构多模态预训练中学习到的跨模态对齐能力。该设定引入两大挑战:跨域参数干扰,即不同数据分布下的参数更新在融合时产生冲突;层间对齐贡献差异,即不同层与投影器对跨模态对齐的贡献不均衡。为此,我们提出PivotMerge框架,包含两个关键组件:共享空间分解与筛选,用于解耦共享对齐模式与域特定差异,并抑制冲突方向;以及对齐引导的分层融合,根据各层对齐贡献分配特定融合权重。我们构建了基于CC12M的系统性后对齐融合评估场景。在多个多模态基准上的大量实验表明,PivotMerge持续优于现有基线,验证了其有效性和泛化能力。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) rely on multimodal pre-training over diverse data sources, where different datasets often induce complementary cross-modal alignment capabilities. Model merging provides a cost-effective mechanism for integrating multiple expert MLLMs with complementary strengths into a unified model. However, existing model merging research mainly focuses on post-finetuning scenarios, leaving the pre-training stage largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from heterogeneous multimodal pre-training. This setting introduces two key challenges: cross-domain parameter interference, where parameter updates learned from different data distributions conflict during merging, and layer-wise alignment contribution disparity, where different layers and projectors contribute unevenly to cross-modal alignment. To address them, we propose \textbf{PivotMerge}, a post-alignment merging framework for cross-modal projectors. PivotMerge incorporates two key components: Shared-space Decomposition and Filtering, which disentangles shared alignment patterns from domain-specific variations and suppresses conflicting directions, and Alignment-guided Layer-wise Merging, which assigns layer-specific merging weights based on differing alignment contributions. We construct systematic CC12M-based post-alignment merging scenarios for evaluation. Extensive experiments on multiple multimodal benchmarks show that PivotMerge consistently outperforms existing baselines, demonstrating its effectiveness and generalization ability.

多模态模型融合对齐机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。