用线性算子统一处理神经网络曲率矩阵，让复杂计算更简单可扩展。
Position: Curvature Matrices Should Be Democratized via Linear Operators
- 通过线性算子接口封装曲率矩阵运算，隐藏实现复杂性。
- 支持大规模神经网络，兼容多种优化与分析任务。
- 适合需要高效曲率计算的研究者与工程开发者。
结构化大矩阵在机器学习中广泛存在，其中曲率矩阵（如海森矩阵）对理解神经网络的损失景观至关重要，可用于二阶优化、不确定性量化、模型剪枝、数据归因等。然而，曲率计算常受自动微分复杂性及各类代理方法（如稀疏性、克罗内克分解）的限制。本文主张，线性算子——即矩阵-向量乘法的接口——可提供通用、可扩展且用户友好的抽象方式来处理曲率矩阵。为此，我们开发了 curvlinops 库，通过统一的线性算子接口提供曲率矩阵。实验表明，该接口能隐藏实现复杂性，简化应用，具备可扩展性和与其他库的互操作性，并可应用于大型神经网络。
原文摘要 · Abstract (English)
Structured large matrices are prevalent in machine learning. A particularly important class is curvature matrices like the Hessian, which are central to understanding the loss landscape of neural nets (NNs), and enable second-order optimization, uncertainty quantification, model pruning, data attribution, and more. However, curvature computations can be challenging due to the complexity of automatic differentiation, and the variety and structural assumptions of curvature proxies, like sparsity and Kronecker factorization. In this position paper, we argue that linear operators -- an interface for performing matrix-vector products -- provide a general, scalable, and user-friendly abstraction to handle curvature matrices. To support this position, we developed $\textit{curvlinops}$, a library that provides curvature matrices through a unified linear operator interface. We demonstrate with $\textit{curvlinops}$ how this interface can hide complexity, simplify applications, be extensible and interoperable with other libraries, and scale to large NNs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。