改进ViT压缩中的费雪近似，提升精度与效率平衡
Mind the Approximation: Fisher-Weighted SVD Compression for ViTs
- 设计结构化费雪近似，保留注意力局部性与梯度依赖
- 在固定计算量下优化各层秩分配，最高提升5.8%准确率
- 无需微调即可适用多种ViT架构，适合部署场景
模型压缩对缓解日益增长的机器学习模型部署挑战至关重要。基于奇异值分解（SVD）的压缩方法在计算效率与模型精度间提供良好权衡，其中费雪加权SVD实现有原则的、损失感知的压缩。然而我们发现，费雪近似保真度提升并不能有效预测视觉变换器（ViTs）压缩后的准确率。为此，我们提出FACTS，一种专为费雪加权SVD压缩ViTs设计的结构化费雪近似，强制实现令牌局部聚合并保留令牌内激活-梯度依赖性。此外，我们引入快速约束秩搜索（CoRS），在固定浮点运算量（FLOP）约束下优化分层秩分配。在ViTs和混合架构上的大量实验表明，FACTS在无需微调的情况下持续改善精度-效率权衡。特别地，在Swin-B上相比最强的SVD基线最高提升5.8个百分点（p.p.）Top-1准确率，且进一步增益来自我们的搜索方法。代码已开源：https://github.com/MoritzTho/FACTS。
原文摘要 · Abstract (English)
Model compression is key to mitigate deployment challenges of ever growing machine learning models. In this area of research, singular value decomposition (SVD)-based compression offers a compelling trade-off between computational efficiency and model accuracy. Fisher-weighted SVD in particular provides principled, loss-aware compression. However, we find that improving the fidelity of Fisher approximation used in the compression is poorly predictive of post-compression accuracy for Vision Transformers (ViTs). Motivated by this observation, we propose FACTS, a structured Fisher Approximation tailored to Compressing ViTs with Fisher-weighted SVD, which enforces token-local aggregation while preserving within-token activation-gradient dependence. Additionally, we introduce a fast Constrained Rank Search (CoRS), that optimizes layer-wise rank allocation while adhering to a fixed floating point operation (FLOP) constraint. Extensive experiments across ViTs and hybrid architectures demonstrate that FACTS consistently improves accuracy-efficiency trade-offs without requiring finetuning. Notably, it outperforms the strongest SVD baseline by up to +5.8 percentage points (p.p.) Top-1 on Swin-B, with further gains driven by our search method. Code is available at https://github.com/MoritzTho/FACTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。