用智能剪枝找到任务真正需要的参数，让大模型更轻更快。
How Many Parameters Does Your Task Really Need? Task Specific Pruning with LLM-Sieve
- 通过输出对齐的投影实现更精准的低秩近似
- 在3.8B到70B模型上剪掉20%-75%参数，损失仅1%-5%精度
- 适合想高效部署或研究模型内部知识分布的研究者
随着大语言模型在资源受限场景中用于特定任务，一个核心问题浮现：完成某项任务究竟需要多少参数？我们提出LLM-Sieve框架，可将大模型裁剪至维持任务性能所需的最小参数子集。方法包含两项创新：(i) 输出对齐的非正交投影，相比传统PCA/SVD能更准确地逼近层输出；(ii) 基于遗传算法的自适应剪枝，自动发现各矩阵的最优剪枝率，揭示任务相关知识分布不均的特性。在3.8B至70B参数模型上，该方法移除20%-75%权重，精度仅下降1%-5%，显著优于现有剪枝方法。除了提升效率，该框架还识别出关键瓶颈矩阵，提示未来模型设计方向。其可无缝集成LoRA微调与量化，兼顾高效部署与对模型知识组织的理解。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) are increasingly deployed for narrow tasks in resource-constrained settings, a central question arises: how much of an LLM is truly necessary for a given task? We present LLM-Sieve, a framework that prunes LLMs down to the minimal parameter subset needed to preserve task performance. Our approach introduces two innovations: (i) output-aligned non-orthogonal projections, which yield more faithful low-rank approximations than traditional PCA/SVD by aligning directly with layer outputs; and (ii) adaptive pruning via a Genetic Algorithm, which automatically discovers matrix-specific pruning levels and exposes the uneven distribution of task-relevant knowledge. Across models from 3.8B to 70B parameters, LLM-Sieve removes 20-75% of weights with only 1-5% accuracy loss-substantially ahead of prior pruning methods. Beyond efficiency, our framework reveals bottleneck matrices that concentrate critical knowledge, suggesting architectural implications for future LLM design. LLM-Sieve integrates seamlessly with LoRA fine-tuning and quantization, enabling both efficient deployment and deeper understanding of knowledge organization in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。