根据激活信息动态调整模型融合权重，提升大模型推理效率与准确率。
Activation-Guided Consensus Merging for Large Language Models
- 基于预训练与微调模型的激活互信息，自适应分配各层融合权重。
- 在Qwen-7B上实现响应长度减少55.3%，推理准确率提升1.3点。
- 无需反向传播或额外训练，适合高效集成多模型能力。
近期研究致力于融合系统2的推理能力与系统1的高效性。尽管现有训练和提示方法在效率与稳定性上存在挑战，模型融合成为整合不同大语言模型（LLMs）能力的有前景策略。然而，传统融合方法常假设各层重要性均等，忽略了神经组件的功能异质性。为此，我们提出激活引导共识融合（ACM），一种即插即用的融合框架，通过预训练与微调模型激活间的互信息确定分层融合系数。ACM无需梯度计算或额外训练，有效保留任务特定能力。在长文本到短文本（L2S）及通用融合任务上的大量实验表明，ACM始终优于所有基线方法。例如，在Qwen-7B模型中，采用ACM的TIES-Merging实现响应长度降低55.3%，同时推理准确率提升1.3点。
原文摘要 · Abstract (English)
Recent research has increasingly focused on reconciling the reasoning capabilities of System 2 with the efficiency of System 1. While existing training-based and prompt-based approaches face significant challenges in terms of efficiency and stability, model merging emerges as a promising strategy to integrate the diverse capabilities of different Large Language Models (LLMs) into a unified model. However, conventional model merging methods often assume uniform importance across layers, overlooking the functional heterogeneity inherent in neural components. To address this limitation, we propose \textbf{A}ctivation-Guided \textbf{C}onsensus \textbf{M}erging (\textbf{ACM}), a plug-and-play merging framework that determines layer-specific merging coefficients based on mutual information between activations of pre-trained and fine-tuned models. ACM effectively preserves task-specific capabilities without requiring gradient computations or additional training. Extensive experiments on Long-to-Short (L2S) and general merging tasks demonstrate that ACM consistently outperforms all baseline methods. For instance, in the case of Qwen-7B models, TIES-Merging equipped with ACM achieves a \textbf{55.3\%} reduction in response length while simultaneously improving reasoning accuracy by \textbf{1.3} points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。