arXiv:2506.06280cs.LGcs.AI2025-06ICML被引 11

提出新方法消除权重矩阵长宽比对谱分析的偏差,提升模型诊断准确性。

Eigenspectrum Analysis of Neural Networks without Aspect Ratio Bias

  • 通过固定长宽比采样子矩阵,修正原权重矩阵的谱分布偏差
  • 在大语言模型剪枝中使困惑度降低17.3%,优于现有方法
  • 适用于图像分类、科学机器学习等多领域,尤其适合层间超参数调优

近年来,通过分析神经网络权重矩阵的特征谱来诊断深度神经网络(DNN)成为研究热点。主流方法是测量权重矩阵经验谱密度(ESD)的重尾程度,以判断模型是否训练充分,并指导分层超参数设置。本文指出,权重矩阵的尺寸与长宽比会引入显著偏差,影响重尾性估计,导致诊断错误和超参数分配失准。为此,我们提出FARMS(Fixed-Aspect-Ratio Matrix Subsampling)方法:通过固定长宽比采样子矩阵,计算其平均谱密度替代原始谱密度,有效缓解长宽比偏差。我们在多种优化技术与应用领域验证该方法,涵盖计算机视觉中的图像分类、科学机器学习(SciML)模型训练及大语言模型(LLM)剪枝。结果表明,尽管方法简单,FARMS能一致提升谱分析精度,实现更有效的分层超参数分配。在一项LLM剪枝实验中,相较当前最优方法,其将LLaMA-7B模型的困惑度降低17.3%。

原文摘要 · Abstract (English)

Diagnosing deep neural networks (DNNs) by analyzing the eigenspectrum of their weights has been an active area of research in recent years. One of the main approaches involves measuring the heavytailness of the empirical spectral densities (ESDs) of weight matrices. This analysis has been shown to provide insights to help diagnose whether a model is well-trained or undertrained, and has been used to guide training methods involving layer-wise hyperparameter assignment. In this paper, we address an often-overlooked challenge in estimating the heavytailness of these ESDs: the impact of the aspect ratio of weight matrices. We demonstrate that matrices of varying sizes (and aspect ratios) introduce a non-negligible bias in estimating the heavytailness of ESDs, leading to inaccurate model diagnosis and layer-wise hyperparameter assignment. To overcome this challenge, we propose FARMS (Fixed-Aspect-Ratio Matrix Subsampling), a method that normalizes the weight matrices by subsampling submatrices with a fixed aspect ratio. Instead of measuring the heavytailness of the original ESD, we measure the average ESD of these subsampled submatrices. We show that this method effectively mitigates the aspect ratio bias. We validate our approach across various optimization techniques and application domains that involve eigenspectrum analysis of weights, including image classification in computer vision (CV) models, scientific machine learning (SciML) model training, and large language model (LLM) pruning. Our results show that despite its simplicity, FARMS uniformly improves the accuracy of eigenspectrum analysis while enabling more effective layer-wise hyperparameter assignment. In one of the LLM pruning experiments, FARMS reduces the perplexity of the LLaMA-7B model by 17.3% when compared with state-of-the-art methods.

谱分析模型诊断大模型剪枝权重分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。