用高效融合方法把多个专业大模型能力整合到一个模型，性能更强还省算力。
InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion
- 通过顶K选择和对齐输出概率，提升多模型知识融合效果
- 在11个任务上超越Qwen和Phi-4等顶尖模型，仅需160小时H800 GPU训练
- 适合需要高性能、低成本集成多领域大模型的开发者
我们提出InfiFusion，一种高效的训练流程，可将多个领域专用的大语言模型（LLMs）融合为单一枢纽模型，充分吸收各源模型的优势。传统融合方法或直接合并参数，或依赖知识蒸馏中僵化的假设，限制了灵活性与效率。InfiFusion通过增强通用逻辑蒸馏（ULD）引入顶K选择与逻辑值标准化，提出两种融合策略：成对融合（InfiFusion$_p$），逐个将源模型知识蒸馏至枢纽模型后合并；统一融合（InfiFusion$_u$），同时将所有源模型知识蒸馏至枢纽模型。InfiFusion在涵盖推理、编程、数学和指令遵循的11个广泛基准测试中均优于当前最优模型，如Qwen-2.5-14B-Instruct和Phi-4。值得注意的是，该方法显著降低计算成本，完整训练仅需160小时H800 GPU，远低于传统大模型训练所需的数百万小时。
原文摘要 · Abstract (English)
We introduce InfiFusion, an efficient training pipeline designed to integrate multiple domain-specialized Large Language Models (LLMs) into a single pivot model, effectively harnessing the strengths of each source model. Traditional fusion methods either merge model parameters directly or rely on knowledge distillation with rigid assumptions, limiting their flexibility and efficiency. InfiFusion overcomes these limitations by enhancing Universal Logit Distillation (ULD) with Top-K selection and Logits Standardization. We propose two fusion strategies: Pairwise Fusion (InfiFusion$_p$), where each source model knowledge is distilled individually into the pivot model followed by merging and Unified Fusion (InfiFusion$_u$), where knowledge from all source models is distilled simultaneously into the pivot model. InfiFusion outperforms the state-of-the-art models, such as Qwen-2.5-14B-Instruct and Phi-4, across 11 widely applied benchmarks covering reasoning, coding, mathematics, and instruction-following tasks. Notably, InfiFusion achieves this superior performance while significantly reduces computational costs, completing full training with only 160 H800 GPU hours compared to the millions typically required for traditional LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。