用神经编码器自动压缩语言模型权重，省去人工设计步骤。
Neural Weight Compression for Language Models
- 将权重压缩建模为神经编码器学习，端到端训练无需手工设计变换。
- 在4-6比特压缩率下表现优异，精度损失小，超越传统方法。
- 适用于多种模型架构，尤其适合追求轻量化部署的研究者。
随着模型规模和部署需求的增长，语言模型权重的高效压缩变得日益重要。然而，现有方法多依赖手工设计的变换和启发式规则，反映出对权重作为数据模态的理解有限。为突破这一局限，我们提出神经权重压缩（NWC）框架，将权重压缩建模为神经编码器学习，并在预训练权重数据集上训练神经编码器。NWC解决了权重压缩中的固有挑战，包括张量异质性以及重建损失与下游性能之间的不匹配问题。实验表明，NWC在精度-压缩率权衡上表现卓越，尤其在4-6比特压缩率区间优势明显，且无需依赖如哈达玛变换等固定手工组件。该方法在多种模型架构（如视觉编码器）中均展现良好泛化能力。分析揭示了熵约束量化与可学习变换在适应权重数据及下游任务中的关键作用。
原文摘要 · Abstract (English)
Efficient compression of language model weights is increasingly critical as model scale and deployment grow. Yet, most existing methods rely on handcrafted transforms and heuristics, reflecting the limited understanding of weights as a data modality. To move beyond this paradigm, we formulate weight compression as neural codec learning and propose Neural Weight Compression (NWC), a framework for training neural codecs on pretrained weight datasets. NWC addresses challenges intrinsic to weight compression, including tensor heterogeneity and the mismatch between reconstruction losses and downstream performance. Experiments show that NWC achieves highly competitive accuracy-compression tradeoffs, with particularly strong results in the 4-6 bit regime, without relying on rigid handcrafted components such as the Hadamard transform. These gains extend to across diverse architectures, e.g., vision encoders. Our analysis highlights the roles of entropy-constrained quantization and learned transforms in adapting compression to weight data and downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。