通过合并相似的前馈子层，实现Transformer模型高效压缩。
Merging Feed-Forward Sublayers for Compressed Transformers
- 不剪枝，而是合并相似的前馈子层以压缩模型。
- 压缩超三分之一子层，仍保持99%原始性能。
- 适合需要轻量化部署的NLP与视觉任务研究者。
随着大型深度学习模型的兴起和普及，高质量压缩技术的需求日益增长，以在不同硬件的内存限制下广泛部署这些模型。本文提出一种新型模型压缩方法：不剪枝重要性较低的参数，而是合并模型中相似的参数组。具体而言，我们选择、对齐并合并Transformer模型中的独立前馈子层，并在语言建模、图像分类和机器翻译任务上测试该方法。实验表明，该方法在合并超过三分之一的前馈子层的同时，性能接近原模型，且优于强基线剪枝方法。例如，在视觉Transformer中可移除超过21%的参数，同时保持99%的原始性能。此外，我们发现部分前馈子层组具有高激活相似性，这可能解释了它们出色的可合并性。
原文摘要 · Abstract (English)
With the rise and ubiquity of larger deep learning models, the need for high-quality compression techniques is growing in order to deploy these models widely. The sheer parameter count of these models makes it difficult to fit them into the memory constraints of different hardware. In this work, we present a novel approach to model compression by merging similar parameter groups within a model, rather than pruning away less important parameters. Specifically, we select, align, and merge separate feed-forward sublayers in Transformer models, and test our method on language modeling, image classification, and machine translation. With our method, we demonstrate performance comparable to the original models while combining more than a third of model feed-forward sublayers, and demonstrate improved performance over a strong layer-pruning baseline. For instance, we can remove over 21% of total parameters from a Vision Transformer, while maintaining 99% of its original performance. Additionally, we observe that some groups of feed-forward sublayers exhibit high activation similarity, which may help explain their surprising mergeability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。