arXiv:2606.25285cs.LGcs.AI2026-06KDD

一次训练实现多稀疏度部署,大幅降低大模型压缩的调优成本。

EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression

论文配图:EPTS: Elastic Post-Training Sparsity for Efficient Large Language Model Compression
图 1 · 摘自论文原文
  • 设计多稀疏层级LoRA机制,实现低稀疏到高稀疏的知识迁移。
  • 引入动态特征混合模块,提升模型对不同稀疏粒度的适应能力。
  • 单次优化即可支持多场景部署,适合资源受限设备的灵活适配。

后训练稀疏化(PTS)已成为压缩大语言模型以实现在资源受限设备上高效部署的关键范式。然而,现有方法通常局限于单一稀疏度优化,每种稀疏度都需要独立且耗时的优化过程。这种僵化模式严重限制了在多样硬件环境中的灵活部署,因适应新稀疏需求需重新优化。为此,我们提出弹性后训练稀疏化(EPTS),一种统一的多稀疏度框架,通过一次优化生成一个可跨多种稀疏配置保持稳健性能的弹性模型。具体地,我们设计了多稀疏层级LoRA(MS-HiLoRA)机制,促进从低稀疏组到高稀疏组的知识继承,有效缓解参数重构的竞争。同时,引入多稀疏特征混合器(MSFM),通过动态融合不同稀疏粒度的特征表示,显著增强模型对剪枝扰动的适应性。在LLaMA与OPT系列模型上的大量实验表明,EPTS在性能上媲美SparseGPT和Wanda等先进方法,同时通过单次优化实现多场景部署,带来显著效率提升。源代码已开源:https://github.com/xuke225/EPTS。

原文摘要 · Abstract (English)

Post-Training Sparsity (PTS) has emerged as a crucial paradigm for compressing Large Language Models to facilitate efficient deployment on resource-constrained devices. However, existing PTS methodologies are typically confined to Single-Sparsity optimization, necessitating a separate, time-consuming optimization session for each specific sparsity level. This rigid paradigm significantly hinders flexible deployment across diverse hardware scenarios, as adapting to a new sparsity requirement mandates a complete re-optimization process. To address these limitations, we propose Elastic Post-Training Sparsity (EPTS), a unified Multi-Sparsity framework that produces a single elastic model capable of maintaining robust performance across diverse sparsity configurations through a one-shot optimization process. Specifically, we design a Multi-Sparsity Hierarchy LoRA (MS-HiLoRA) mechanism that facilitates knowledge inheritance from low- to high-sparsity groups, effectively mitigating the competition for parameter reconstruction. Furthermore, we introduce a Multi-Sparsity Feature Mixer (MSFM), which significantly enhances the model's adaptability to pruning perturbations by dynamically fusing feature representations of varying sparsity granularities. Extensive experiments on LLaMA and OPT families demonstrate that EPTS achieves competitive performance compared to state-of-the-art methods like SparseGPT and Wanda, while offering significant efficiency gains by enabling multi-scenario deployment from a single optimization. our source code is available at https://github.com/xuke225/EPTS.

模型压缩稀疏化LoRA高效部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。