不依赖数据和微调，通过合并神经元压缩模型
Forget the Data and Fine-Tuning! Just Fold the Network to Compress
- 通过聚类合并跨层相似神经元，实现无数据压缩
- 在高稀疏度下性能优于现有无数据方法
- 适合资源受限环境部署大模型
我们提出模型折叠（model folding），一种无需训练数据和微调的新型无数据模型压缩技术。该方法通过合并跨层结构相似的神经元显著减小模型规模，同时利用k-means聚类保留数据统计特性，并采用新提出的无数据技术防止方差坍缩或爆炸。理论分析与实验证明，该方法在标准基准（包括ResNet18和LLaMA-7B）上性能接近数据驱动压缩方法，尤其在高稀疏度下优于近期提出的无数据方法。该技术对大规模模型压缩尤为有效，适用于资源受限环境中的部署。
原文摘要 · Abstract (English)
We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。