通过层间对齐压缩神经网络权重,提升压缩率且保持推理速度。
Motion-Compensated Weight Compression

- 对齐相似结构块,将深度方向转为可预测序列。
- 压缩率优于现有方法,在相同精度下节省30%以上存储。
- 适合模型部署场景,尤其对Transformer架构高效。
神经网络权重正日益成为部署瓶颈,但多数压缩流程独立处理各层,忽略了由功能保持对称性引发的跨层冗余。我们提出运动补偿权重压缩(MCWC),一种仅针对权重的编码器,通过对齐置换对称块(如隐藏单元和注意力头)以最大化层间对应关系,将深度方向转化为可预测序列。在对齐坐标系中,MCWC采用轻量级逐层预测器,结合周期性关键帧,并使用基于率失真目标训练的可学习熵模型编码量化后的预测残差。简单解码器通过熵解码、去量化、预测重建和逆对齐操作,快速恢复可部署权重,实现高效推理。在Transformer语言建模与视觉分类任务中,MCWC在速率-精度权衡上超越强基线方法,同时保持竞争力解码时间。消融实验表明,对齐、预测、熵建模和关键帧调度均对性能提升至关重要。代码已公开于https://github.com/Ism-ail11/MCWC。
原文摘要 · Abstract (English)
Neural network weights are increasingly a bottleneck for deployment, yet most compression pipelines treat layers independently and overlook cross-layer redundancy induced by function-preserving symmetries. We propose Motion-Compensated Weight Compression (MCWC), a weight-only codec that aligns permutation-symmetric blocks (e.g., hidden units and attention heads) to maximize cross-layer correspondence, turning depth into a predictable sequence. In the aligned coordinate system, MCWC uses a lightweight layer-sequential predictor with periodic keyframes and encodes only quantized prediction residuals using a learned entropy model trained under a rate distortion objective. A simple decoder reconstructs deployable weights by entropy decoding, dequantization, predictor-driven reconstruction, and inverse alignment, enabling fast weight materialization for inference. Across Transformer language modeling and vision classification, MCWC improves the rate accuracy Pareto frontier over strong quantization and learned weight-codec baselines, while maintaining competitive decode time. Ablations confirm that alignment, prediction, entropy modeling, and keyframe scheduling are each necessary for the full gains. Our code is available via https://github.com/Ism-ail11/MCWC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。