用参数激活对齐度量预测模型泛化能力，无需验证集。
A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio
- 通过权重谱与激活谱的重叠度定义对齐比，衡量参数与激活匹配程度。
- 在深度学习中，对齐比下降时模型从记忆转向泛化，且可预测有效学习维度。
- 训练中实时计算，无额外开销，适合监控大模型过拟合风险。
本文研究参数化理论中的对数对齐比（LAR），将其重新定义为权重谱 $p$（矩阵奇异值平方的归一化分布）与激活谱 $q$（输入在奇异方向上投影平方的归一化分布）之间的重叠。在两种不同设置下，未嵌入的LAR能追踪模型从记忆到泛化的转变，通过捕捉 $p$ 与 $q$ 的分布扩散过程实现。在“领悟”（grokking）场景中，LAR可预测所学函数的有效维度：$k \approx n^{2(1- ext{LAR})}$，其中 $n$ 为矩阵输入维度。在30亿参数语言模型预训练中，其相对于非过拟合基线的偏差能跟踪泛化差距，且过拟合逼近时其下降速率加快。LAR可在前向传播中实时计算，计算开销极小，且无需预留验证数据。
原文摘要 · Abstract (English)
We study the log-alignment ratio (LAR), a measure of parameter-activation alignment, introduced in parameterization theory. We reformulate it as the overlap between a weight spectrum $p$ of the normalized squared singular values of a matrix and an activation spectrum $q$ of the normalized squared projections of inputs onto its singular directions. We show that unembedding LAR tracks the transition between memorization and generalization in two different settings by capturing the spread of $p$ and $q$ during training. In grokking, LAR predicts the effective dimension of the learned function: $k \approx n^{2(1-\text{LAR})}$, where $n$ is the input dimension of the matrix. In 3B-parameter language model pre-training, its deviation from a non-overfitting baseline tracks the generalization gap, and its rate of decline increases as overfitting approaches. LAR is computable from quantities available during the forward pass with negligible computational overhead, and requires no held-out validation data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。