通过分层激活重要性，让模型合并更精准提升特定任务表现。
LEWIS (LayEr WIse Sparsity) -- A Training Free Guided Model Merging Approach
- 根据激活值动态调整各层重要性，指导合并时的稀疏策略。
- 在代码指令和数学求解任务上分别提升4%和11.3%性能。
- 无需训练与数据，适合希望优化特定任务的模型合并场景。
随着专用大语言模型日益普及，模型合并方法被用于在不需额外数据或训练的情况下，将多个模型融合为单一多任务模型。然而，当目标是提升某个特定任务基准上的下游性能时，现有方法效果有限。本文提出LEWIS（Layer Wise Sparsity），一种基于激活的分层重要性引导的模型合并框架，通过校准数据集识别关键层，在任务向量剪枝过程中动态调整层级稀疏度。该方法在保留关键层任务特异性知识的同时，使合并后模型在与校准数据集相似的基准上表现最优。实验表明，使用LEWIS合并得到的代码指令遵循和数学求解模型，性能分别提升4%和11.3%，显著优于无引导的均匀稀疏合并方法。
原文摘要 · Abstract (English)
As specialized large language models (LLMs) become increasingly prevalent, model merging methods are being used to combine them to create a single multi-task model without requiring any additional data or training. However, these approaches fall short when the objective of merging is to increase the downstream model's performance on a particular task-specific benchmark. In this work, we propose LEWIS (Layer Wise Sparsity), a guided model-merging framework that uses activation-based layer importance to dynamically adjust layer-wise task-vector sparsity required for the merge process. LEWIS uses a calibration dataset to prioritize critical layers during the task-vector pruning process required for model merging. This approach guides existing merging methods by preserving essential layer-wise task-specific knowledge while ensuring the merged model performs the best at benchmarks resembling the calibration dataset. Our experiments demonstrate the effectiveness of LEWIS with performance improvements of code instruction-following and math-solving models created through model merging up to 4 percent and 11.3 percent, respectively, outperforming unguided data-less model merging approaches that use uniform-sparsity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。