用随机矩阵压缩神经网络激活值，节省内存并实时监控梯度。
Randomized Matrix Sketching for Neural Network Training and Gradient Monitoring
- 通过自适应秩调整的指数移动平均矩阵，压缩层激活数据。
- 在MNIST等数据集上实现精度与内存消耗的可控权衡。
- 适合需要低内存训练或实时梯度分析的研究场景。
神经网络训练依赖反向传播计算梯度，但存储各层激活值的内存开销带来显著可扩展性挑战。本文首次将控制理论中的矩阵随机化压缩技术应用于神经网络激活值，实现反向传播中梯度的高效重建。该方法基于近期动态优化问题的矩阵压缩框架，利用三个互补的压缩矩阵，通过指数移动平均（EMA）动态维护，并支持自适应秩调整，在内存效率与近似质量间自动平衡。在MNIST、CIFAR-10及物理信息神经网络上的实证评估表明，可实现精度与内存占用的可控权衡。我们在MNIST上展示了梯度监控应用，证明压缩激活值能以极低内存开销实现梯度范数的实时追踪。结果表明，压缩激活存储为高效神经网络训练与分析提供了可行路径。
原文摘要 · Abstract (English)
Neural network training relies on gradient computation through backpropagation, yet memory requirements for storing layer activations present significant scalability challenges. We present the first adaptation of control-theoretic matrix sketching to neural network layer activations, enabling memory-efficient gradient reconstruction in backpropagation. This work builds on recent matrix sketching frameworks for dynamic optimization problems, where similar state trajectory storage challenges motivate sketching techniques. Our approach sketches layer activations using three complementary sketch matrices maintained through exponential moving averages (EMA) with adaptive rank adjustment, automatically balancing memory efficiency against approximation quality. Empirical evaluation on MNIST, CIFAR-10, and physics-informed neural networks demonstrates a controllable accuracy-memory tradeoff. We demonstrate a gradient monitoring application on MNIST showing how sketched activations enable real-time gradient norm tracking with minimal memory overhead. These results establish that sketched activation storage provides a viable path toward memory-efficient neural network training and analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。