arXiv:2602.15200cs.LG2026-02

用校准数据优化正交分解,实现Transformer模型无训练压缩

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

  • 基于校准数据估计稀疏权重分解,使用正交字典实现闭式更新
  • 单步求解系数,避免迭代优化,压缩后精度损失小
  • 动态分配各层压缩率,适合多种模型和量化场景

Transformer模型的训练后压缩通常依赖截断奇异值分解(SVD),但共享单一子空间会降低精度。稀疏字典学习提供更灵活的子空间并集表示,但现有方法常需迭代更新字典与系数。本文提出COMPOT(校准优化矩阵Procrustes正交化用于Transformer压缩),一种无需训练的压缩框架,利用小规模校准数据估计稀疏权重分解。COMPOT采用正交字典,使字典更新可闭式求解,系数计算为单步稀疏编码,无需迭代优化。针对不同层对压缩敏感度差异,在全局压缩预算下引入一次性动态分配策略,自适应调整各层压缩率。在多种架构和任务上的实验表明,COMPOT在质量-压缩权衡上优于强基线低秩与稀疏方法,且与训练后量化兼容,支持极端压缩。代码已公开。

原文摘要 · Abstract (English)

Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available $\href{https://github.com/mts-ai/COMPOT}{here}$.

模型压缩稀疏分解正交化量化兼容

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。