用奇异学习理论提升神经网络压缩评估的严谨性
Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory
- 基于局部学习系数估算模型复杂度
- 复杂度与压缩率呈紧密线性关系
- 适合研究模型压缩极限的学者
我们利用奇异学习理论，将最小描述长度（MDL）原理扩展至神经网络等奇异性模型。在Pythia系列数据集上，通过量化、分解等多种压缩技术进行大量实验发现，基于局部学习系数（LLC）的复杂度估计值与模型可压缩性高度相关，某些情况下甚至呈线性关系。该结果为严格评估模型压缩上限提供了可行路径。
原文摘要 · Abstract (English)
We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。