arXiv:2509.23587cs.LGcs.NA2025-09

同时估算低秩与对角成分,提升大型矩阵近似精度

Sketching Low-Rank Plus Diagonal Matrices

  • 联合估计低秩与对角结构,避免分步近似的误差累积
  • 在合成LoRD矩阵上实现高精度恢复,优于分步方法
  • 适用于深度学习海森矩阵等大规模高保真近似场景

许多机器学习和科学计算任务涉及高维线性算子,仅可通过昂贵的矩阵-向量乘法访问。近期的压缩方法可从少量矩阵-向量乘法中构建低秩或对角近似,显著提升速度与可扩展性,但因假设结构简化而引入近似误差。本文提出SKETCHLORD,同时估计低秩与对角成分,针对更广泛的低秩加对角(LoRD)线性算子。理论上与实证均表明,该联合估计优于任何串行变体(先对角后低秩或反之)。我们将SKETCHLORD建模为凸优化问题,得到可扩展算法。在合成(近似)LoRD矩阵上的全面实验验证其能准确恢复此类结构。该方法可作为结构化近似工具箱的重要补充,尤其适用于深度学习海森矩阵等需高保真近似的大型算子。

原文摘要 · Abstract (English)

Many relevant machine learning and scientific computing tasks involve high-dimensional linear operators accessible only via costly matrix-vector products. In this context, recent advances in sketched methods have enabled the construction of *either* low-rank *or* diagonal approximations from few matrix-vector products. This provides great speedup and scalability, but approximation errors arise due to the assumed simpler structure. This work introduces SKETCHLORD, a method that simultaneously estimates both low-rank *and* diagonal components, targeting the broader class of Low-Rank *plus* Diagonal (LoRD) linear operators. We demonstrate theoretically and empirically that this joint estimation is superior also to any sequential variant (diagonal-then-low-rank or low-rank-then-diagonal). Then, we cast SKETCHLORD as a convex optimization problem, leading to a scalable algorithm. Comprehensive experiments on synthetic (approximate) LoRD matrices confirm SKETCHLORD's performance in accurately recovering these structures. This positions it as a valuable addition to the structured approximation toolkit, particularly when high-fidelity approximations are desired for large-scale operators, such as the deep learning Hessian.

矩阵近似低秩凸优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。