用几何维度预测模型性能，无需标签也能判断好坏。
On the Relationship Between Representation Geometry and Generalization in Deep Neural Networks
- 用无监督几何指标有效维度衡量表示结构。
- 该维度与准确率相关性高达0.75，超越模型大小。
- 对图像、文本任务均适用，且可反向验证因果关系。
我们研究了表示几何与深度神经网络性能之间的关系。分析52个预训练的ImageNet模型（覆盖13种架构），发现有效维度——一种无监督几何度量——能强预测准确率。控制模型容量后，输出有效维度的偏相关系数达0.75（p < 10⁻¹⁰），总压缩比为-0.72。该结论在ImageNet和CIFAR-10上复现，并推广至NLP：有效维度对SST-2/MNLI上的8个编码器模型和AG News上的15个解码器模型均有预测力（r=0.69，p=0.004），而模型大小无显著相关性（r=0.07）。我们建立双向因果关系：通过加噪破坏几何结构会导致准确率下降（r=-0.94，p < 10⁻⁹）；通过PCA提升几何结构可在不同架构间维持准确率（仅损失0.03个百分点，保留95%方差）。该关系对噪声类型不敏感，高斯、均匀、丢弃、椒盐噪声均保持| r | > 0.90。结果表明，有效维度可提供跨领域、无标签的预测与因果信息。
原文摘要 · Abstract (English)
We investigate the relationship between representation geometry and neural network performance. Analyzing 52 pretrained ImageNet models across 13 architecture families, we show that effective dimension -- an unsupervised geometric metric -- strongly predicts accuracy. Output effective dimension achieves partial r=0.75 ($p < 10^(-10)$) after controlling for model capacity, while total compression achieves partial r=-0.72. These findings replicate across ImageNet and CIFAR-10, and generalize to NLP: effective dimension predicts performance for 8 encoder models on SST-2/MNLI and 15 decoder-only LLMs on AG News (r=0.69, p=0.004), while model size does not (r=0.07). We establish bidirectional causality: degrading geometry via noise causes accuracy loss (r=-0.94, $p < 10^(-9)$), while improving geometry via PCA maintains accuracy across architectures (-0.03pp at 95% variance). This relationship is noise-type agnostic -- Gaussian, Uniform, Dropout, and Salt-and-pepper noise all show $|r| > 0.90$. These results establish that effective dimension provides domain-agnostic predictive and causal information about neural network performance, computed entirely without labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。