用最优传输方法实现大模型向小模型的知识迁移。
Transport and Merge: Cross-Architecture Merging for Large Language Models
- 基于最优传输对齐激活值，建立异构模型间神经元对应关系。
- 仅需少量输入即可实现大模型到小模型的有效知识迁移。
- 适合低资源语言与垂直领域的小模型优化任务。
大型语言模型（LLMs）通过扩大模型容量和训练数据获得强大能力，但许多实际部署仍依赖于小规模、低资源数据训练或微调的模型。这一差距促使人们需要将大模型中的知识高效迁移到小模型中。尽管模型合并提供了一种有效的迁移机制，但现有方法通常假设模型架构兼容，无法直接将高资源大模型的知识转移到异构的小模型中。本文提出一种基于最优传输（OT）的跨架构合并框架，通过对齐激活值来推断异构模型间的跨神经元对应关系，进而利用生成的传输计划指导权重空间融合，仅需少量输入即可实现从高资源模型到低资源目标模型的有效知识迁移。在低资源语言和特定领域上的大量实验表明，该方法持续优于目标模型。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data, yet many real-world deployments rely on smaller models trained or adapted from low-resource data. This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets. While model merging provides an effective transfer mechanism, most existing approaches assume architecture-compatible models and therefore cannot directly transfer knowledge from large high-resource LLMs to heterogeneous low-resource targets. In this work, we propose a cross-architecture merging framework based on optimal transport (OT) that aligns activations to infer cross-neuron correspondences between heterogeneous models. The resulting transport plans are then used to guide direct weight-space fusion, enabling effective high-resource to low-resource transfer using only a small set of inputs. Extensive experiments across low-resource languages and specialized domains demonstrate consistent improvements over target models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。