arXiv:2603.29466cs.LGcs.AI2026-03被引 1

仅用一次前向传播,快速估算大模型预测不确定性。

An Isotropic Approach to Efficient Uncertainty Quantification with Gradient Norms

  • 基于梯度范数和各向同性假设,从单次推理中估算不确定性。
  • 在合成任务上与蒙特卡洛方法结果高度一致,且模型越大越准。
  • 发现参数级不确定性与自评方法信号不同,适用于冲突答案场景。

现有神经网络预测不确定性量化方法要么计算成本过高,要么需访问训练数据(通常不可得)。本文通过两项近似提出轻量替代方案:一阶泰勒展开将不确定性表示为预测梯度与参数协方差的函数;参数协方差采用各向同性假设。二者结合后,得到的表征型不确定性即为梯度范数平方,随机型不确定性为点预测的伯努利方差,仅需对未修改的预训练模型进行一次前向-反向传播即可获得。我们通过实证表明,基于非训练数据构建的协方差估计会引入结构偏差,而各向同性假设可避免该问题;同时大型网络的谱性质理论支持该近似在规模上的合理性。在合成问题上与参考马尔可夫链蒙特卡洛估计对比显示强对应关系,且随模型增大而提升。进一步用于大语言模型问答任务中的答案正确性预测,发现联合估计在TruthfulQA上取得最高均值AUROC(因存在合理答案间的真冲突),但在TriviaQA的事实召回任务中接近随机水平,表明参数级不确定性捕捉的是与自评估不同的信号。

原文摘要 · Abstract (English)

Existing methods for quantifying predictive uncertainty in neural networks are either computationally intractable for large language models or require access to training data that is typically unavailable. We derive a lightweight alternative through two approximations: a first-order Taylor expansion that expresses uncertainty in terms of the gradient of the prediction and the parameter covariance, and an isotropy assumption on the parameter covariance. Together, these yield epistemic uncertainty as the squared gradient norm and aleatoric uncertainty as the Bernoulli variance of the point prediction, from a single forward-backward pass through an unmodified pretrained model. We justify the isotropy assumption by showing that covariance estimates built from non-training data introduce structured distortions that isotropic covariance avoids, and that theoretical results on the spectral properties of large networks support the approximation at scale. Validation against reference Markov Chain Monte Carlo estimates on synthetic problems shows strong correspondence that improves with model size. We then use the estimates to investigate when each uncertainty type carries useful signal for predicting answer correctness in question answering with large language models, revealing a benchmark-dependent divergence: the combined estimate achieves the highest mean AUROC on TruthfulQA, where questions involve genuine conflict between plausible answers, but falls to near chance on TriviaQA's factual recall, suggesting that parameter-level uncertainty captures a fundamentally different signal than self-assessment methods.

不确定性大模型梯度评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。