提出新算法,在内存有限下高效估算超大矩阵对数行列式。
Determinant Estimation under Memory Constraints and Neural Scaling Laws
- 基于分块LDL分解的层级算法,避免存储完整矩阵。
- 仅用极小数据量即可准确估算神经正切核对数行列式,提速10万倍。
- 适用于大规模神经网络模型分析,尤其适合计算资源受限场景。
大规模正定矩阵的对数行列式计算在机器学习中至关重要。传统方法因三次复杂度已难处理,而现代应用中存储矩阵本身更成内存瓶颈。本文提出一种基于分块LDL分解的新型层级算法,适用于内存受限环境下的大规模对数行列式计算。当矩阵高度病态时,完整矩阵计算可能不可行,这在大规模核矩阵(如训练于大数据集的神经网络经验神经正切核,NTK)中尤为显著。在测试误差符合神经缩放律的假设下,我们发现伪行列式比值满足幂律关系,由此推导出相应缩放律。该方法使我们仅需极小数据子集即可准确估计NTK对数行列式,在实验中实现约10万倍加速,且精度优于现有近似方法。利用此技术,我们成功估算出此前因规模过大、计算需求过高而无法处理的密集矩阵对数行列式。
原文摘要 · Abstract (English)
Calculating or accurately estimating log-determinants of large positive definite matrices is of fundamental importance in many machine learning tasks. While its cubic computational complexity can already be prohibitive, in modern applications, even storing the matrices themselves can pose a memory bottleneck. To address this, we derive a novel hierarchical algorithm based on block-wise computation of the LDL decomposition for large-scale log-determinant calculation in memory-constrained settings. In extreme cases where matrices are highly ill-conditioned, accurately computing the full matrix itself may be infeasible. This is particularly relevant when considering kernel matrices at scale, including the empirical Neural Tangent Kernel (NTK) of neural networks trained on large datasets. Under the assumption of neural scaling laws in the test error, we show that the ratio of pseudo-determinants satisfies a power-law relationship, allowing us to derive corresponding scaling laws. This enables accurate estimation of NTK log-determinants from a tiny fraction of the full dataset; in our experiments, this results in a $\sim$100,000$\times$ speedup with improved accuracy over competing approximations. Using these techniques, we successfully estimate log-determinants for dense matrices of extreme sizes, which were previously deemed intractable and inaccessible due to their enormous scale and computational demands.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。