arXiv:2608.13596cs.LGcs.AI2026-08

不靠对齐,用激活信号剪枝大模型,小模型性能提升明显。

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

论文配图:Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning
图 1 · 摘自论文原文
  • 基于激活强度筛选大模型关键组件,剪裁后注入小模型。
  • 小模型平均准确率从55.5%提升至60.6%,多项任务超70%。
  • 适合资源受限场景下快速迁移大模型能力,无需训练。

异构模型融合旨在整合任务、初始化、架构或规模不同的模型。本文研究一个未被充分探索的跨尺度场景:在存在显著架构差异的情况下,利用更强的大型捐赠者模型提升小型接收者语言模型的性能。我们探讨是否能在无需逐神经元语义对齐的前提下实现有效能力迁移。基于观察——将大模型截断至小模型架构并引入微小混合权重即可改善接收者表现,我们提出激活引导剪枝融合框架(APM)。APM在捐赠者上构建任务相关的激活图,选择关键层、隐藏维度、注意力头和MLP神经元进行剪枝,形成适配接收者架构的捐赠者片段,并以极小插值系数注入原接收者。该方法将捐赠者视为功能组件的集中源,而非需精确结构移植的对象。在16个涵盖推理、数学、代码生成、指令遵循和分类的任务中,接收者平均准确率从55.5%提升至60.6%;RTE准确率从64.3%升至82.3%,QNLI从52.3%升至65.7%,BoolQ从70.8%升至79.2%。对注入比例与多阶段融合的分析表明,激活引导提取能提升可转移片段质量,同时维持低比例融合特性。结果表明,当捐赠者贡献足够集中且被精心选择时,跨尺度异构融合可在无显式语义对齐下成功实现。

原文摘要 · Abstract (English)

Heterogeneous model fusion seeks to combine models that differ in tasks, initializations, architectures, or scales. We study an underexplored cross-scale setting: improving a small recipient language model with a stronger donor despite substantial architectural mismatch. We ask whether useful capabilities can be transferred without explicit neuron-wise semantic alignment. Building on the observation that truncating a large model to a smaller architecture and injecting it with a tiny mixing weight can already improve the recipient, we propose Activation-Prune-Merge (APM), an activation-guided framework for cross-scale fusion. APM constructs task-conditioned activation maps on the donor, selects salient layers, hidden dimensions, attention heads, and MLP neurons to prune it to the recipient architecture, and injects the resulting donor slice into the original recipient using a micro interpolation coefficient. This formulation treats the donor as a source of concentrated functional components rather than requiring precise structural transplantation. Across 16 benchmarks spanning reasoning, mathematics, code generation, instruction following, and classification, APM improves the overall average accuracy from 55.5% to 60.6% over the original 3B recipient. RTE accuracy increases from 64.3% to 82.3%, QNLI from 52.3% to 65.7%, and BoolQ from 70.8% to 79.2%. Analyses of injection ratios and sequential multi-stage fusion further suggest that activation-guided extraction improves the quality of the transferable donor slice while preserving the small-ratio fusion regime. These results provide evidence that cross-scale heterogeneous fusion can succeed without explicit semantic alignment when the donor contribution is sufficiently concentrated and carefully selected.

模型压缩知识迁移剪枝小模型增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。