用稀疏字典学习实现大模型压缩，精度更高且支持稀疏计算。
CoSpaDi: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning
- 用字典学习替代单一子空间，让不同权重列选不同原子组合。
- 在20%-40%压缩率下，比SVD和结构化剪枝更优，困惑度更低。
- 无需训练，兼容稀疏-密集计算与后训练量化，适合部署优化。
后训练大模型压缩通常依赖低秩近似，强制投影矩阵的所有列共享单一低维子空间。我们提出CoSpaDi，一种无需训练的压缩框架，通过稀疏字典学习将单子空间假设替换为子空间并集模型。CoSpaDi将每个权重矩阵分解为稠密字典与列稀疏系数，使不同列在相同存储预算下可选择不同字典原子子集。为保持模型行为，我们利用校准激活将功能重构转化为标准字典学习问题。在Llama和Qwen模型上，CoSpaDi在20%-40%压缩率下，相比基于SVD和结构化剪枝的基线，在准确率-压缩与困惑度-压缩权衡上均有提升，同时天然支持稀疏-密集计算及稀疏系数的后训练量化。
原文摘要 · Abstract (English)
Post-training LLM compression often relies on low-rank approximations, which force all columns of a projection matrix to share a single low-dimensional subspace. We propose CoSpaDi, a training-free compression framework that replaces this single-subspace assumption with a union-of-subspaces model via sparse dictionary learning. CoSpaDi factorizes each weight matrix into a dense dictionary and column-sparse coefficients, allowing different columns to select different subsets of dictionary atoms at the same storage budget. To preserve model behavior, we use calibration activations to transform functional reconstruction into a standard dictionary learning problem. Across Llama and Qwen models, CoSpaDi improves accuracy--compression and perplexity--compression trade-offs over SVD-based and structured pruning baselines at 20--40\% compression ratios, while naturally supporting sparse--dense computation and post-training quantization of sparse coefficients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。