用神经压缩距离测试大模型分类能力，发现压缩率不能预测准确率。
Neural Normalized Compression Distance and the Disconnect Between Compression and Classification
- 将大模型转为无损压缩器，构建神经化归一化压缩距离
- 压缩率高的模型分类准确率反而更低，与预期相反
- 挑战了'更好压缩=更好分类'的直觉，适合研究模型本质的读者
在信息论中，预测分类与压缩被认为是内在相关的概念。许多深度学习方法被解释为学习某种压缩形式，且更好的压缩被认为能带来更好的性能。本文通过归一化压缩距离（NCD）检验这一假设，该方法直接依赖压缩来衡量序列间的相似性，从而实现最近邻分类。我们把流行的大型语言模型（LLMs）转化为无损压缩器，构建神经化NCD，并将其与gzip等经典通用算法进行比较。结果发现，分类准确率无法仅由压缩率预测，还存在其他未被现有理论解释的异常现象。这表明，我们对神经网络‘压缩’的含义以及有效分类所需条件的理解仍不充分。
原文摘要 · Abstract (English)
It is generally well understood that predictive classification and compression are intrinsically related concepts in information theory. Indeed, many deep learning methods are explained as learning a kind of compression, and that better compression leads to better performance. We interrogate this hypothesis via the Normalized Compression Distance (NCD), which explicitly relies on compression as the means of measuring similarity between sequences and thus enables nearest-neighbor classification. By turning popular large language models (LLMs) into lossless compressors, we develop a Neural NCD and compare LLMs to classic general-purpose algorithms like gzip. In doing so, we find that classification accuracy is not predictable by compression rate alone, among other empirical aberrations not predicted by current understanding. Our results imply that our intuition on what it means for a neural network to ``compress'' and what is needed for effective classification are not yet well understood.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。