arXiv:2606.28432stat.MLcs.AI2026-06

研究量化导致的参数模型信息矩阵谱变化,揭示其可作为大模型部署时的运行监控指标。

Spectral Perturbation of the Empirical Fisher Information Matrix under Weight Quantization

  • 基于Weyl不等式建立量化噪声下的最大特征值下界,证明其在主导阶上严格增大。
  • 实测显示4比特量化后该特征值比全精度估计值高约244倍,与理论预测一致。
  • 提出可计算的近似方法,适合用于大模型量化后的实时健康监测。

我们研究了参数化统计模型在两种结构化扰动下的经验费雪信息矩阵(FIM)谱扰动:输入偏离参考分布集合,以及模型参数的有限精度(量化)扰动。针对第一类扰动,在最大特征值λ_max的局部曲率单调性假设下,证明偏离参考流形会使其相对校准基线显著升高(命题3.2),并解释该假设必要性。主要结果为通过Weyl不等式获得方向性特征值扰动下界,表明量化噪声扰动下λ_max不低于未扰动值,且在温和通用条件下主导阶严格大于原值(定理4.3)。我们给出两种λ_max的可计算近似——一种启发式,一种具严格双侧界——以及增强状态空间中基于阈值划分的完备性结果。这些结果支持将σ_t = λ_max(F_t)/λ_base作为部署语言模型的运行时监控统计量:量化结果解释了我们的一项经验观察——4比特量化模型的校准阈值约为全精度预估的244倍,单次测量而非闭式解。报告12个模型、n=1,080条轨迹的支持性测量,讨论各结果的适用范围与局限,并指出闭式预测量化膨胀幅度仍为开放问题。

原文摘要 · Abstract (English)

We study the spectral perturbation of the empirical Fisher Information Matrix (FIM) of a parametric statistical model under two structured perturbations: departure of the input from a reference (in-distribution) ensemble, and finite-precision (quantized) perturbation of the model's parameters. For the first, under an explicit local curvature-monotonicity hypothesis on the dominant eigenvalue lambda_max of the FIM, we show departure from a reference manifold provably elevates lambda_max relative to a calibration baseline (Proposition 3.2), and discuss why this hypothesis is required, since curvature need not increase monotonically under every perturbation. Our principal result is a directional eigenvalue perturbation bound, via Weyl's inequality, showing lambda_max under a quantization noise perturbation is lower bounded by its unperturbed value up to a third-order remainder, and, under a mild genericity condition, strictly exceeds it at leading order (Theorem 4.3). We give two tractable approximations to lambda_max -- one heuristic, one with a rigorous two-sided bound -- and a completeness result for a threshold-based partition of an augmented state space. These results motivate using sigma_t = lambda_max(F_t)/lambda_base as a runtime monitoring statistic for deployed language models: the quantization result offers a mechanism for an empirical observation of our own, where a calibration threshold for this statistic was approximately 244 times larger than a preliminary full-precision estimate on a 4-bit quantized model, a single measurement rather than a value derived in closed form. We report supporting measurements (twelve models, n=1,080 trajectories) broadly consistent with our predictions, discuss the scope and limitations of every result, and state as an open problem the closed-form prediction of the quantization inflation magnitude our bound does not supply.

量化信息矩阵大模型监控谱分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。