统一压缩方法的数学框架,让大模型变小更高效。
Big2Small: A Unifying Neural Network Framework for Model Compression
- 用测度论构建统一框架,所有压缩技术本质是正则化神经网络。
- 无需数据训练,通过隐式网络编码大模型权重,压缩比达10倍以上。
- 适合追求高压缩率且无数据可用的模型部署场景。
随着基础模型的发展,模型压缩已成为关键需求。现有压缩方法如低秩分解、剪枝、量化、遍历动态系统和知识蒸馏,基于不同启发式策略。为将该领域从碎片化提升为严谨体系,本文基于测度论构建统一数学框架,证明各类压缩技术在数学上等价于带正则化的神经网络。在此基础上,提出可实验验证的数据无关压缩框架Big2Small,将隐式神经表示(INRs)从数据域映射至网络参数域。Big2Small训练紧凑型INRs以编码大模型权重,并在推理时重建权重。为提升重建精度,引入异常值感知预处理以应对极端权重值,以及频率感知损失函数以保留高频细节。图像分类与分割实验表明,Big2Small在准确率和压缩比方面均达到当前最优水平。
原文摘要 · Abstract (English)
With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed \textit{Big2Small}, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. \textit{Big2Small} trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that \textit{Big2Small} achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。