模型折叠比剪枝更优，理论与实证均支持其在无重训练下压缩模型
Cut Less, Fold More: Model Compression through the Lens of Projection Geometry
- 将剪枝和折叠统一为正交投影，从几何角度分析压缩机制
- 折叠在中高压缩率下准确率更高，参数重构误差更小
- 适合追求高效部署且无需调参的开发者使用
在不进行微调的情况下压缩神经网络对大规模部署至关重要。本文从投影几何视角研究无校准压缩：结构化剪枝是轴对齐投影，而模型折叠通过权重重聚类实现低秩投影。我们将其形式化为正交算子，并证明在秩距离为1时，折叠的参数重建误差必然更小，在温和平滑性假设下，函数扰动也小于剪枝。我们在超过1000个检查点上评估了ResNet18、PreActResNet18、ViT-B/32、CLIP ViT-B/32在CIFAR-10和ImageNet-1K上的表现，涵盖多种训练超参数（优化器、学习率、数据增强、正则化、尖锐感知训练），以及多个60M和130M参数的LLaMA家族模型在C4上的训练结果。结果显示，折叠通常在压缩后取得更高准确率，尤其在中高压缩率时优势明显；特定训练设置下差距缩小甚至反转。结果表明，折叠是一种几何感知、无需校准的压缩方法，理论严谨且实践中常优于剪枝。
原文摘要 · Abstract (English)
Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。