用随机方法快速计算神经网络核的特征，提升分析效率。
Fast Neural Tangent Kernel Alignment, Norm and Effective Rank via Trace Estimation
- 通过迹估计实现无矩阵计算，加速核矩阵分析。
- 在小样本下，单向自动微分比传统方法快数倍。
- 适合需要快速评估模型动态的研究者使用。
神经正切核（NTK）描述了模型状态在梯度下降过程中的演化。对递归结构而言，完整计算NTK矩阵通常不可行。本文提出一种无矩阵视角，利用迹估计快速分析经验性有限宽度NTK。该方法可高效计算NTK的迹、Frobenius范数、有效秩和对齐度。基于Hutch++迹估计器，我们提供了具有严格收敛保证的数值算法。此外，由于NTK结构特性，仅需前向或反向自动微分即可计算迹，无需双模式。在样本量少时，这类单向估计器表现优于Hutch++，尤其当模型状态与参数量差距较大时。总体表明，无矩阵随机方法可实现数量级加速，显著提升NTK分析与应用速度。
原文摘要 · Abstract (English)
The Neural Tangent Kernel (NTK) characterizes how a model's state evolves over Gradient Descent. Computing the full NTK matrix is often infeasible, especially for recurrent architectures. Here, we introduce a matrix-free perspective, using trace estimation to rapidly analyze the empirical, finite-width NTK. This enables fast computation of the NTK's trace, Frobenius norm, effective rank, and alignment. We provide numerical recipes based on the Hutch++ trace estimator with provably fast convergence guarantees. In addition, we show that, due to the structure of the NTK, one can compute the trace using only forward- or reverse-mode automatic differentiation, not requiring both modes. We show these so-called one-sided estimators can outperform Hutch++ in the low-sample regime, especially when the gap between the model state and parameter count is large. In total, our results demonstrate that matrix-free randomized approaches can yield speedups of many orders of magnitude, leading to faster analysis and applications of the NTK.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。