提出重要性感知的稀疏化方法，显著提升大模型压缩与合并效率。
ImPart: Importance-Aware Delta-Sparsification for Improved Model Compression and Merging in LLMs
- 基于SVD动态调整奇异向量稀疏率，按重要性保留关键参数。
- 在相同性能下压缩比达基线2倍，高稀疏下仍保持任务知识。
- 适用于模型压缩、量化和合并，适合资源受限场景部署。
随着任务特定大型语言模型的普及，增量压缩成为缓解部署多个模型资源压力的有效手段。以往的增量稀疏化方法或随机移除参数，或在奇异值分解（SVD）后直接截断奇异向量，前者忽略参数重要性，后者粒度粗糙。本文提出ImPart，一种新的重要性感知增量稀疏化方法。利用SVD动态调整不同奇异向量的稀疏比例，依据其重要性保留关键任务知识，即使在高稀疏率下依然有效。实验表明，ImPart在相同性能水平下实现比基线高出2倍的压缩比。结合现有方法后，其在增量量化与模型合并任务上均达到新最优表现。
原文摘要 · Abstract (English)
With the proliferation of task-specific large language models, delta compression has emerged as a method to mitigate the resource challenges of deploying numerous such models by effectively compressing the delta model parameters. Previous delta-sparsification methods either remove parameters randomly or truncate singular vectors directly after singular value decomposition (SVD). However, these methods either disregard parameter importance entirely or evaluate it with too coarse a granularity. In this work, we introduce ImPart, a novel importance-aware delta sparsification approach. Leveraging SVD, it dynamically adjusts sparsity ratios of different singular vectors based on their importance, effectively retaining crucial task-specific knowledge even at high sparsity ratios. Experiments show that ImPart achieves state-of-the-art delta sparsification performance, demonstrating $2\times$ higher compression ratio than baselines at the same performance level. When integrated with existing methods, ImPart sets a new state-of-the-art on delta quantization and model merging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。