通过融合模型输出分布，实现大模型高效知识共享与能力保留。
MoD: A Distribution-Based Approach for Merging Large Language Models
- 直接操作输出概率分布，而非模型权重进行合并
- 在数学推理任务上显著优于现有合并方法
- 适合需要多任务能力且资源受限的部署场景
大型语言模型（LLMs）催生了众多专用、任务特定的变体。然而，这些独立模型的维护与部署在资源利用和运行效率方面面临巨大挑战。本文提出一种基于分布的模型合并框架——混合分布（Mixture of Distributions, MoD），直接在模型输出的概率分布层面进行合并，而非传统权重平均方式。与现有方法相比，MoD能有效保留各模型的专长能力，同时实现跨任务的知识高效共享。在基于Qwen2.5模型的数学推理基准测试中，实验表明MoD在多个评测集上均显著超越现有模型合并技术。相关代码、数据及实验材料已公开于https://github.com/knovel-eng/mod。
原文摘要 · Abstract (English)
Large language models (LLMs) have enabled the development of numerous specialized, task-specific variants. However, the maintenance and deployment of these individual models present substantial challenges in terms of resource utilization and operational efficiency. In this work, we propose the \textit{Mixture of Distributions (MoD)} framework, a novel approach for merging LLMs that operates directly on their output probability distributions, rather than on model weights. Unlike traditional weight-averaging methods, MoD effectively preserves the specialized capabilities of individual models while enabling efficient knowledge sharing across tasks. Through extensive experimentation on mathematical reasoning benchmarks using Qwen2.5 models, we demonstrate that MoD significantly outperforms existing model merging techniques across multiple benchmarks. All code, data, and experimental materials are published at https://github.com/knovel-eng/mod.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。