通过逐维动态调整稀疏度，实现大模型极致压缩且保持性能
TRIM: Achieving Extreme Sparsity with Targeted Row-wise Iterative Metric-driven Pruning
- 按输出维度逐行动态分配稀疏度，而非统一设置
- 80%稀疏下Qwen2.5-14B困惑度降低48%，OPT-13B超90%
- 适合追求极致模型压缩的部署场景
大型语言模型因规模庞大带来显著计算与内存挑战，剪枝成为高效部署的关键。现有单次剪枝方法通常对各层或层内应用统一稀疏度，导致高稀疏率下性能下降严重。本文提出TRIM（目标导向的逐行迭代度量驱动剪枝），为每层中每个输出维度（行）分配可变稀疏度。TRIM通过质量度量引导的迭代调整过程，优化维度级稀疏分配，重点降低输出间质量保留方差，以保护关键信息。该方法可无缝集成现有层级剪枝策略。在多种LLM系列（Qwen2.5、LLaMA-2、OPT）及不同稀疏度下的困惑度与零样本任务评估中，TRIM取得新最优结果并提升稳定性。例如，在80%稀疏度下，相比基线方法，Qwen2.5-14B困惑度降低48%，OPT-13B降低超过90%。结论表明，细粒度的维度级稀疏自适应对突破大模型压缩极限至关重要。代码已开源：https://github.com/flobk/TRIM
原文摘要 · Abstract (English)
Large Language Models (LLMs) present significant computational and memory challenges due to their extensive size, making pruning essential for their efficient deployment. Existing one-shot pruning methods often apply uniform sparsity constraints across layers or within each layer, resulting in suboptimal performance, especially at high sparsity ratios. This work introduces TRIM (Targeted Row-wise Iterative Metric-driven pruning), a novel approach that applies varying sparsity ratios to individual output dimensions (rows) within each layer. TRIM employs an iterative adjustment process guided by quality metrics to optimize dimension-wise sparsity allocation, focusing on reducing variance in quality retention across outputs to preserve critical information. TRIM can be seamlessly integrated with existing layer-wise pruning strategies. Our evaluations on perplexity and zero-shot tasks across diverse LLM families (Qwen2.5, LLaMA-2, and OPT) and sparsity levels demonstrate that TRIM achieves new state-of-the-art results and enhances stability. For instance, at 80% sparsity, TRIM reduces perplexity by 48% for Qwen2.5-14B and over 90% for OPT-13B compared to baseline methods. We conclude that fine-grained, dimension-wise sparsity adaptation is crucial for pushing the limits of extreme LLM compression. Code available at: https://github.com/flobk/TRIM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。