不训练即可实现高效低秩模型剪枝，适应不同任务需求。
Diet Your LLM: Dimension-wise Global Pruning of LLMs via Merging Task-specific Importance Score
- 用少量样本分析各任务激活强度，通过投票生成全局剪枝掩码。
- 在Gemma-2 2B模型上，20%稀疏度下平均准确率提升近10%。
- 无需训练或预计算，适合快速部署的模型压缩场景。
大语言模型（LLM）虽性能卓越，但其庞大规模给实际部署带来挑战。结构化剪枝通过移除整个维度或层提供解决方案，但现有方法存在权衡：通用方法无法适配任务需求，而任务感知方法需昂贵训练以学习适应性。本文提出DIET（基于任务重要性得分融合的维度级全局剪枝），一种无需训练的结构化剪枝方法，兼具维度级粒度与任务感知选择。DIET仅需每任务100个样本分析激活幅度，再通过多数投票构建单一全局掩码，避免大规模预计算或训练开销。在七个零样本基准上使用Gemma-2 2B和9B模型的实验表明，该方法有效；例如，在Gemma-2 2B上20%稀疏度下，相比当前最优结构化剪枝方法，平均准确率提升近10%。此优势在多种稀疏度和模型规模下均保持稳定，使DIET成为实用且鲁棒的结构化LLM剪枝方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated remarkable capabilities, but their massive scale poses significant challenges for practical deployment. Structured pruning offers a promising solution by removing entire dimensions or layers, yet existing methods face critical trade-offs: task-agnostic approaches cannot adapt to task-specific requirements, while task-aware methods require costly training to learn task adaptability. We propose DIET (Dimension-wise global pruning of LLMs via merging Task-wise importance scores), a training-free structured pruning method that combines dimension-level granularity with task-aware selection. DIET profiles activation magnitudes across tasks using only 100 samples per task, then applies majority voting to construct a single global mask. DIET does not require large costs from pre-computation or training. Experiments on seven zero-shot benchmarks using Gemma-2 2B and 9B models demonstrate the effectiveness of DIET; for example, at 20% sparsity on Gemma-2 2B, DIET achieves near 10% average accuracy improvement, compared to previous state-of-the-art structured pruning methods. This advantage persists across various sparsity levels and model scales, positioning DIET as a practical and robust choice for structured LLM pruning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。