用随机矩阵理论评估深度网络权重奇异值截断阈值的合理性
Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory
- 将权重矩阵分解为信号与噪声分量，用随机矩阵理论定阈值去噪
- 提出余弦相似度指标衡量截断后奇异向量与原矩阵的一致性
- 适合研究模型压缩、低秩近似或权重分析的科研人员
本研究评估基于随机矩阵理论的奇异值阈值在深度神经网络权重矩阵低秩近似中的有效性。每个权重矩阵被建模为信号矩阵与噪声矩阵之和，通过设定阈值移除与噪声相关的奇异值以实现低秩近似。为评估该阈值的合理性，提出一种基于信号矩阵与原始权重矩阵奇异向量间余弦相似度的评价指标，并用于数值实验，比较两种阈值估计方法的表现。
原文摘要 · Abstract (English)
This study evaluates thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices. Each weight matrix is modeled as the sum of signal and noise matrices. The low-rank approximation is obtained by removing noise-related singular values using a threshold based on random matrix theory. To assess the adequacy of this threshold, we propose an evaluation metric based on the cosine similarity between the singular vectors of the signal and original weight matrices. The proposed metric is used in numerical experiments to compare two threshold estimation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。