用克罗内克积压缩GPT-2，模型更小但性能更强。
Krony-PT: GPT2 compressed with Kronecker Products
- 用克罗内克积压缩Transformer前馈层权重，降低参数量。
- 最小81M参数模型在多个数据集上优于DistilGPT2。
- 新初始化方法提升压缩后模型性能，适合轻量化部署。
我们提出Krony-PT，一种基于克罗内克积的GPT-2压缩方法，重点针对每个Transformer块的前馈层权重矩阵进行系统性压缩。引入改进的Van Loan分解初始化新克罗内克因子，并提出一种基于剪枝的初始化技术。该方法将原始124M参数的GPT-2压缩为80M至96M的不同版本。其中81M模型在所有标准语言建模数据集上的下一个词预测任务中表现优于DistilGPT2，且在显著更小的克罗内克压缩模型中达到竞争性或相当的性能。
原文摘要 · Abstract (English)
We introduce Krony-PT, a compression technique for GPT-2 based on Kronecker products. We specifically target the feed-forward weights of each transformer block, and systematically compress the feed-forward layer matrices to various degrees. We introduce a modified Van Loan decomposition to initialize new Kronecker factors, and also propose a new pruning-based initialization technique. Our method compresses the original 124M-parameter GPT-2 to various smaller models, ranging from 80M to 96M. Our 81M model variant outperforms DistilGPT2 on next-token prediction across all standard language modeling datasets, and shows competitive or comparable performance with significantly larger Kronecker-based compressions of GPT-2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。