用参数扰动下的预测方差统一分析模型鲁棒性,可无标签评估训练效果。
TPV: Parameter Perturbations Through the Lens of Test Prediction Variance
- 提出测试预测方差(TPV),衡量模型对参数微小变化的敏感度。
- TPV在过参数化时从训练集即可推断测试集表现,且与测试损失高度相关。
- 无需标签即可实现剪枝和模型选择,适用于小样本与迁移学习场景。
我们提出测试预测方差(TPV)——即训练模型输出对参数扰动的一阶敏感度——作为后训练鲁棒性分析的统一框架。TPV是完全无标签的量,其迹形式将模型几何结构与扰动机制解耦,使SGD噪声、标签噪声、量化和剪枝均可在同一视角下分析。理论证明,在过参数化极限下,训练集上的TPV收敛至测试集对应值,不依赖泛化性能,这是首个表明局部参数扰动下预测方差可仅由训练输入推断的结果。实验显示该稳定性在极低宽度下依然成立。此外,TPV与测试损失强相关,催生了两项应用:基于TPV几何匹配的无标签剪枝方法JBR,达到当前最优水平;以及面向分布内和迁移学习的训练集模型选择信号。代码已开源。
原文摘要 · Abstract (English)
We introduce test prediction variance (TPV)--the first-order sensitivity of a trained model's outputs to parameter perturbations--as a unifying framework for analyzing post-training robustness. TPV is a fully label-free object whose trace form separates the geometry of the trained model from the specific perturbation mechanism, placing SGD noise, label noise, quantization, and pruning under a single lens. The resulting expressions recover the wide-minima hypothesis for SGD and quantization noise, and yield a distinct Jacobian-spectral characterization for label noise connecting label-noise TPV with benign overfitting in nonlinear networks. Theoretically, we prove that training-set TPV converges to its test-set counterpart in the overparameterized limit, irrespective of generalization performance, providing the first result that prediction variance under local parameter perturbations can be inferred from training inputs alone. Empirically, this stability holds far more broadly, including at very low widths. Further, TPV correlates well with test loss, enabling practical applications: JBR, a label-free pruning criterion derived from TPV geometry matching state-of-the-art baselines; and training-set based model selection signal for in-distribution and transfer learning scenarios. Code available at github.com/devansharpit/TPV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。