arXiv:2602.11008cs.LGcs.AI2026-02被引 1

ROCKET通过优化分配与快速稀疏化,实现无训练高效模型压缩。

ROCKET: Rapid Optimization via Calibration-guided Knapsack Enhanced Truncation for Efficient Model Compression

  • 将每层压缩量视为多重选择背包问题求解,最小化重建误差
  • 30%压缩率下保留超90%性能,无需微调
  • 仅用小校准集完成稀疏化,避免反向传播与迭代优化

我们提出ROCKET,一种无需训练的模型压缩方法,在因子分解、结构化稀疏与动态压缩基线中达到最优表现。在全局压缩预算下,其包含两项核心创新:首先,将层间压缩分配建模为多选背包问题,选择各层最优压缩率以最小化总重建误差并满足目标模型大小;其次,提出一种单步稀疏矩阵分解方法,受字典学习启发:仅使用少量校准集,基于激活-权重敏感度稀疏化权重系数,并通过最小二乘法闭式更新字典,完全跳过迭代优化、稀疏编码与反向传播。ROCKET在不同模型架构下均优于现有压缩方法,在20%-50%压缩率时表现突出。特别地,30%压缩率下性能保留超过90%,且无需微调。若加入轻量微调,效果显著提升:例如将Qwen3-14B压缩至8B参数后,仅用3000万令牌微调,性能几乎媲美原版Qwen3-8B。ROCKET代码位于github.com/mts-ai/ROCKET/tree/main。

原文摘要 · Abstract (English)

We present ROCKET, a training-free model compression method that achieves state-of-the-art performance in comparison with factorization, structured-sparsification and dynamic compression baselines. Operating under a global compression budget, ROCKET comprises two key innovations: First, it formulates layer-wise compression allocation as a multi-choice knapsack problem, selecting the optimal compression level for each layer to minimize total reconstruction error while adhering to a target model size. Second, it introduces a single-step sparse matrix factorization inspired by dictionary learning: using only a small calibration set, it sparsifies weight coefficients based on activation-weights sensitivity and then updates the dictionary in closed form via least squares bypassing iterative optimization, sparse coding, or backpropagation entirely. ROCKET consistently outperforms existing compression approaches across different model architectures at 20-50\% compression rates. Notably, it retains over 90\% of the original model's performance at 30\% compression without any fine-tuning. Moreover, when applying a light fine-tuning phase, recovery is substantially enhanced: for instance, compressing Qwen3-14B to an 8B-parameter model and healing it with just 30 million tokens yields performance nearly on par with the original Qwen3-8B. The code for ROCKET is at github.com/mts-ai/ROCKET/tree/main.

模型压缩无训练稀疏化量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。