自动检测梯度稀疏性，让复杂导数计算快上百倍。
Sparser, Better, Faster, Stronger: Sparsity Detection for Efficient Automatic Differentiation
- 基于操作符重载，自动识别局部与全局稀疏模式
- 真实场景下加速最高达千倍，突破导数计算成本瓶颈
- 无需改代码，适合科学计算、图神经网络等场景
从隐式微分到概率建模，雅可比和海森矩阵在机器学习中有广泛应用，但因其计算成本过高而常被忽略。值得庆幸的是，这些矩阵通常具有稀疏性，可通过自动稀疏微分（ASD）加速。本文提出新型稀疏性检测方法，解决了此前制约ASD性能的瓶颈。该方法基于操作符重载，可自动识别局部与全局稀疏结构，支持灵活的索引集表示，完全无需修改用户代码，兼容现有机器学习框架。更重要的是，其高性能使雅可比和海森矩阵在以往被认为过高的规模下仍可高效计算。在科学机器学习、图神经网络和优化任务的真实案例中，实现了最高达三个数量级的加速。特别地，使用该系统后，单次计算的ASD速度已超越标准AD，无需稀疏性检测或矩阵着色的开销摊销。
原文摘要 · Abstract (English)
From implicit differentiation to probabilistic modeling, Jacobian and Hessian matrices have many potential use cases in Machine Learning (ML), but they are viewed as computationally prohibitive. Fortunately, these matrices often exhibit sparsity, which can be leveraged to speed up the process of Automatic Differentiation (AD). This paper presents advances in sparsity detection, previously the performance bottleneck of Automatic Sparse Differentiation (ASD). Our implementation of sparsity detection is based on operator overloading, able to detect both local and global sparsity patterns, and supports flexible index set representations. It is fully automatic and requires no modification of user code, making it compatible with existing ML codebases. Most importantly, it is highly performant, unlocking Jacobians and Hessians at scales where they were considered too expensive to compute. On real-world problems from scientific ML, graph neural networks and optimization, we show significant speed-ups of up to three orders of magnitude. Notably, using our sparsity detection system, ASD outperforms standard AD for one-off computations, without amortization of either sparsity detection or matrix coloring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。