提出稳定高效的上下文感知低秩近似框架，解决神经网络压缩中的数值不稳定性问题。
COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation
- 无需矩阵求逆，基于稳定分解避免数值误差
- 支持超大校准矩阵、近似奇异输入等困难场景
- 理论证明收敛性并给出误差界，适合模型压缩与微调
近期研究表明，上下文感知低秩近似是压缩和微调大规模神经网络的有效工具。此类方法通过输入激活矩阵加权范数，显著优于无权重情况。然而，现有神经网络方法因依赖显式计算和求逆的格拉姆矩阵，存在数值不稳定性，可能导致近似质量下降或矩阵奇异。本文提出一种全新的无求逆正则化框架，完全基于稳定分解，克服了先前方法的数值缺陷。该方法可应对三种挑战：(1) 校准矩阵超出GPU内存容量；(2) 输入激活矩阵接近奇异；(3) 数据不足导致无法唯一近似。针对第三种情况，我们证明解的收敛性并推导出显式误差界。
原文摘要 · Abstract (English)
Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximation, a norm is weighted by a matrix of input activations, significantly improving metrics over the unweighted case. Nevertheless, existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion. We demonstrate that this can degrade the approximation quality or cause numerically singular matrices. To address these limitations, we propose a novel inversion-free regularized framework that is based entirely on stable decompositions and overcomes the numerical pitfalls of prior art. Our method can handle possible challenging scenarios: (1) when calibration matrices exceed GPU memory capacity, (2) when input activation matrices are nearly singular, and even (3) when insufficient data prevents unique approximation. For the latter, we prove that our solution converges to a desired approximation and derive explicit error bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。