提出统一的低秩模型压缩排名方法,提升压缩效率与性能
UniRank: Unified Rank Allocation for Low-Rank LLM Compression

- 基于局部能量比与全局功能重要性双重评分分配秩
- 一阶段压缩可降低困惑度达50%,优于均匀与启发式分配
- 支持多种模型结构,无需额外微调,适合部署优化场景
低秩分解是大语言模型压缩的有前景范式,但秩分配仍具挑战:人工规则泛化性差,学习型方法计算开销大。本文将全局低秩分配建模为排序-截断流程,通过双准则评分每个奇异分量:局部奇异能量比衡量分解参数矩阵内的重要性,全局功能重要性(以输入输出余弦相似度衡量)评估分解模块的功能意义。几何解释与实验证明高输入输出余弦相似度与低有效秩强相关。此外,提出保持秩的微调策略,在分解权重上直接进行LoRA微调,避免传统合并流程中因重新截断导致的信息损失。实验表明,该方法在不同分解方案、模型规模和架构下均持续提升性能;例如在一次性压缩且无需进一步微调时,困惑度较均匀与启发式分配基线最高降低50%。代码将开源于https://github.com/EIT-NLP/LLM-Pruning。
原文摘要 · Abstract (English)
Low-rank decomposition serves as a promising compression paradigm for large language models, however, rank allocation remains challenging: manual rules lack generalizability, and learning-based approaches incur heavy computational overhead. To address these issues, we formulate global low-rank allocation as a sorting-and-truncation pipeline, and score each singular component via dual criteria: \textbf{Local} singular energy ratio that quantifies the intrinsic importance within the decomposed parameter matrix and \textbf{Global} functional importance (measured by input-output cosine similarity) that evaluates the functional significance of decomposed modules. We verify the strong correlation between high input-output cosine similarity and low effective rank through geometric interpretation and experimental validation. Furthermore, we propose rank-preserving fine-tuning, which performs direct LoRA tuning on decomposed weights and avoids extra information loss caused by re-truncation in conventional merging pipelines. Empirical results confirm that our method delivers sustained performance enhancements when combined with models featuring distinct decomposition schemes, model sizes and architectural designs, e.g. in one-shot compression without further fine-tuning, our method reduces perplexity by up to 50\% compared with uniform and heuristic allocation baselines. Code will be available at https://github.com/EIT-NLP/LLM-Pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。