用关键奇异值替代冗余层，实现高效压缩且性能损失小
GRASP: Replace Redundant Layers with Adaptive Singular Parameters for Efficient Model Compression
- 通过梯度归因识别关键奇异分量，自适应保留重要信息
- 在20%压缩比下保持原模型90%的性能表现
- 适合追求高效推理的LLM部署场景
近期研究表明大型语言模型中存在大量功能冗余层，可通过移除这些层降低推理成本。然而，盲目剪枝常导致性能显著下降。本文提出GRASP（基于梯度的自适应奇异参数保留）压缩框架，通过在小规模校准数据集上进行梯度归因，自适应识别并保留敏感的奇异值。该方法将冗余层替换为极少量参数，实现高效压缩同时保持强性能，且开销极低。在多个LLM上的实验表明，GRASP持续优于现有压缩方法，在20%压缩比下达到原模型90%的性能。
原文摘要 · Abstract (English)
Recent studies have demonstrated that many layers are functionally redundant in large language models (LLMs), enabling model compression by removing these layers to reduce inference cost. While such approaches can improve efficiency, indiscriminate layer pruning often results in significant performance degradation. In this paper, we propose GRASP (Gradient-based Retention of Adaptive Singular Parameters), a novel compression framework that mitigates this issue by preserving sensitivity-aware singular values. Unlike direct layer pruning, GRASP leverages gradient-based attribution on a small calibration dataset to adaptively identify and retain critical singular components. By replacing redundant layers with only a minimal set of parameters, GRASP achieves efficient compression while maintaining strong performance with minimal overhead. Experiments across multiple LLMs show that GRASP consistently outperforms existing compression methods, achieving 90% of the original model's performance under 20% compression ratio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。