利用旋转对称性让大模型权重压缩省出3-5%的存储空间。
Getting Free Bits Back from Rotational Symmetries in LLMs
- 基于比特回溯编码，挖掘Transformer权重的旋转对称性。
- 在不损失性能前提下，压缩率提升3-5%，跨模型通用。
- 适合关注模型压缩与高效部署的研究者和工程师。
当前神经网络权重压缩方法（如分解、剪枝、量化、通道模拟）常忽略网络内部的固有对称性，导致冗余信息占用额外比特。本文提出一种基于比特回溯编码的存储格式，可更高效地存储具有旋转对称性的Transformer权重，相比传统数组布局，在相同浮点精度下实现压缩优化。我们在经SliceGPT（Ashkboos et al., 2024）剪枝的大语言模型上进行评估，结果表明，在不同模型规模与架构下，总比特使用量可免费减少3-5%，且在一定数值精度范围内不影响模型性能。
原文摘要 · Abstract (English)
Current methods for compressing neural network weights, such as decomposition, pruning, quantization, and channel simulation, often overlook the inherent symmetries within these networks and thus waste bits on encoding redundant information. In this paper, we propose a format based on bits-back coding for storing rotationally symmetric Transformer weights more efficiently than the usual array layout at the same floating-point precision. We evaluate our method on Large Language Models (LLMs) pruned by SliceGPT (Ashkboos et al., 2024) and achieve a 3-5% reduction in total bit usage for free across different model sizes and architectures without impacting model performance within a certain numerical precision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。