对比三种压缩方法对LLaMA-2-7B的性能影响，发现困惑度不等于任务表现。
Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models
- 比较幅度剪枝、SparseGPT和Wanda在50%稀疏下的表现
- 稀疏模型困惑度不变但下游任务性能大幅下降
- 建议用JS散度评估压缩后行为变化，需针对性校准数据
大语言模型（LLMs）虽能力强但计算开销大，推动高效压缩技术发展。本研究评估了主流压缩方法——幅度剪枝、SparseGPT和Wanda对LLaMA-2-7B模型的影响，重点关注模型规模缩减、下游任务性能与校准数据的作用之间的权衡。结果表明，尽管SparseGPT和Wanda在50%稀疏下仍能保持困惑度，但在下游任务上出现显著退化，揭示了仅以困惑度作为评估指标的不足。为此，我们引入焦恩-申松（JS）散度作为更全面的评价指标，以捕捉压缩后模型行为的细微变化。进一步证明，针对特定任务的校准数据相比通用校准数据可显著提升压缩模型的下游性能。该研究强调需采用多样化评估指标并谨慎选择校准数据，以充分理解大模型压缩的复杂性及其在实际应用中的影响。
原文摘要 · Abstract (English)
Large language models (LLMs) offer powerful capabilities but incur substantial computational costs, driving the need for efficient compression techniques. This study evaluates the impact of popular compression methods - Magnitude Pruning, SparseGPT, and Wanda - on the LLaMA-2-7B model, focusing on the trade-offs between model size reduction, downstream task performance, and the role of calibration data. Our findings reveal that while SparseGPT and Wanda preserve perplexity even at 50% sparsity, they suffer significant degradation on downstream tasks, highlighting the inadequacy of perplexity as the sole evaluation metric. To address this, we introduce Jensen-Shannon (JS) Divergence as a more comprehensive metric that captures nuanced changes in model behavior post-compression. We further demonstrate that task-specific calibration data significantly enhances the downstream performance of compressed models compared to general calibration data. This research underscores the necessity for diverse evaluation metrics and careful calibration data selection to fully understand the complexities of LLM compression and its implications for practical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。