提出统一框架,自动选择最佳分解秩,实现高效模型压缩。
Unified Framework for Pre-trained Neural Network Compression via Decomposition and Optimized Rank Selection
- 联合分解与秩选择,用复合损失优化压缩配置。
- 在无需额外数据下,自动搜索连续空间最优秩,降低计算开销。
- 压缩后模型性能接近原始模型,适合移动端部署。
尽管复杂神经网络具有高精度,但其巨大的计算资源需求给移动设备和嵌入式系统等资源受限场景的部署带来挑战。压缩算法通过减小模型规模和计算量来应对这一问题。其中基于张量分解的方法理论合理且有效,但在分解秩的选择上存在困难。本文提出一种统一框架,同时进行分解与秩选择,采用带秩约束的复合压缩损失。方法包含在连续空间中的自动秩搜索,可高效识别预训练模型的最佳秩配置,无需额外训练数据,且搜索阶段计算开销低。结合后续微调步骤,该方法使高度压缩模型的性能保持与原始模型相当。在多个基准数据集和模型上,通过全面分析验证了方法的有效性。
原文摘要 · Abstract (English)
Despite their high accuracy, complex neural networks demand significant computational resources, posing challenges for deployment on resource constrained devices such as mobile phones and embedded systems. Compression algorithms have been developed to address these challenges by reducing model size and computational demands while maintaining accuracy. Among these approaches, factorization methods based on tensor decomposition are theoretically sound and effective. However, they face difficulties in selecting the appropriate rank for decomposition. This paper tackles this issue by presenting a unified framework that simultaneously applies decomposition and rank selection, employing a composite compression loss within defined rank constraints. Our method includes an automatic rank search in a continuous space, efficiently identifying optimal rank configurations for the pre-trained model by eliminating the need for additional training data and reducing computational overhead in the search step. Combined with a subsequent fine-tuning step, our approach maintains the performance of highly compressed models on par with their original counterparts. Using various benchmark datasets and models, we demonstrate the efficacy of our method through a comprehensive analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。