不训练也不对齐,仅靠加权平均就能合并不同规模大模型。
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

- 通过轻量级维度适配后按比例插值融合模型。
- 小比例插值可提升性能,但接近平衡时易崩溃。
- 适合快速验证异构模型融合效果的场景。
能否在不训练、不语义对齐的情况下,直接通过加权平均合并参数空间差异巨大的大语言模型?现有异构融合方法通常引入蒸馏、适配器、学习的隐空间、路由或特征对齐,但尚未验证更简单的方案是否可行。本文通过无训练的维度适应与比例控制插值重新审视该问题。在并集式融合中,将小模型扩展至大模型参数空间;在交集式融合中,将大模型截断至小模型空间。在涵盖数学推理、代码生成、语言理解、常识推理、知识和指令遵循的Qwen系列模型对与基准测试中,确定性扩展基本保持源模型功能,小比例插值可借助互补能力超越强源检查点。然而,接近平衡的插值常导致性能崩溃,任务级结果呈现此消彼长现象,即某些能力提升伴随其他能力退化。结果表明,结合轻量维度适配与精细比例控制的简单参数平均,是异构大模型融合的惊人强基线,提示复杂融合方法的上限也可能受直接加权融合的极限所制约。
原文摘要 · Abstract (English)
Can large language models with substantially different parameter spaces be merged by direct weighted averaging, without training or semantic alignment? Existing heterogeneous fusion methods typically introduce distillation, adapters, learned latent spaces, routing, or feature alignment, leaving open whether a simpler recipe can work for genuinely different billion-parameter checkpoints. We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled interpolation. In union-style merging, we expand the smaller model into the larger parameter space; in intersection-style merging, we truncate the larger model into the smaller parameter space. Across Qwen-family model pairs and benchmarks covering mathematical reasoning, code generation, language understanding, commonsense reasoning, knowledge, and instruction following, deterministic expansion largely preserves the source model function, and small-ratio interpolation can improve over strong source checkpoints by transferring complementary capabilities. However, near-balanced interpolation often collapses, and task-level results reveal a seesaw effect in which gains on some capabilities coexist with regressions on others. These results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, is a surprisingly strong baseline for heterogeneous LLM merging, suggesting that the limits of direct weighted fusion may also bound what more complex heterogeneous merging methods can achieve at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。