解决不同架构大模型融合难题,让异构模型协同工作。
Can Heterogeneous Language Models Be Fused?

- 通过模块结构匹配实现跨架构知识迁移
- 在多种异构场景下性能超越现有方法
- 适合需融合多来源大模型的实用场景
模型融合旨在将多个专家模型合并为单一模型,继承其互补优势且无需集成推理开销。近期研究表明,当所有源模型同源(如基于同一预训练主干)时,融合效果良好;但在开放模型生态中,有用专家常基于不同架构(如Llama、Qwen、Mistral),导致直接权重融合因架构不匹配、潜在空间错位和跨源冲突而失效。本文提出 exttt{HeteroFusion},包含两个核心组件:基于拓扑结构对齐,通过匹配功能模块而非原始张量坐标实现跨架构知识转移;冲突感知去噪,抑制融合过程中的不兼容或噪声信号。我们进一步提供理论分析,证明保留目标适配器基底并预测结构化更新可实现稳定、良态的迁移过程。在异构迁移、多源融合、抗噪声源及跨家族泛化等场景中, exttt{HeteroFusion}持续优于强基线方法。
原文摘要 · Abstract (English)
Model merging aims to integrate multiple expert models into a single model that inherits their complementary strengths without incurring the inference-time cost of ensembling. Recent progress has shown that merging can be highly effective when all source models are \emph{homogeneous}, i.e., derived from the same pretrained backbone and therefore share aligned parameter coordinates or compatible task vectors. Yet this assumption is increasingly unrealistic in open model ecosystems, where useful experts are often built on different families such as Llama, Qwen, and Mistral. In such \emph{heterogeneous} settings, direct weight-space fusion becomes ill-posed due to architectural mismatch, latent basis misalignment, and amplified cross-source conflict. We address this problem with \texttt{HeteroFusion} for heterogeneous language model fusion, which consists of two key components: topology-based alignment that transfers knowledge across heterogeneous backbones by matching functional module structures instead of raw tensor coordinates, and conflict-aware denoising that suppresses incompatible or noisy transfer signals during fusion. We further provide analytical justification showing that preserving the target adapter basis while predicting structured updates leads to a stable and well-conditioned transfer process. Across heterogeneous transfer, multi-source fusion, noisy-source robustness, and cross-family generalization settings, \texttt{HeteroFusion} consistently outperforms strong merging, fusion, and ensemble baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。