通过分片特征蒸馏实现快速神经网络张量化压缩。
Fast Tensorization of Neural Networks via Slice-wise Feature Distillation
- 将网络切分为独立模块，逐片张量化以保留原始中间特征。
- ResNet-34上实现近无损压缩，优化速度更快，数据需求更低。
- 适合大规模模型压缩，尤其适用于分布式训练场景。
我们提出一种基于分片特征蒸馏的可扩展神经网络压缩框架。与依赖昂贵全局微调的传统张量分解方法不同，该方法将网络划分为单个层或层块（如卷积层或MLP）组成的片段，或连续几层的小组合，并独立对每个片段进行张量化，以复现原预训练模型的中间表示。这种模块化策略提升了准确率恢复效果，降低了数据需求，支持高效并行优化。在ResNet-34上的实验显示，相比传统全局张量化，该方法在中等压缩率下实现了显著性能提升，接近无损压缩且优化更快。GPT-2 XL的结果进一步验证了该方法的可扩展性及其在大规模模型中的适用性，尤其在分布式设置中表现优异。
原文摘要 · Abstract (English)
We propose a scalable tensorization framework for neural network compression based on slice-wise feature distillation. Unlike conventional tensor decomposition methods that rely on costly global finetuning, our approach decomposes the network into slices consisting of either individual layers or blocks (e.g., convolutional layers or MLPs), or small groups of consecutive layers, and tensorizes each slice independently to reproduce the intermediate representations of the original pretrained model. This modular strategy improves accuracy recovery, reduces data requirements, and enables efficient parallel optimization. Experiments on ResNet-34 show significant gains over conventional global tensorization, achieving near-lossless compression at moderate compression rates with faster optimization. Results on GPT-2 XL further demonstrate the scalability of the method and its applicability to large-scale models, particularly in distributed settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。