用热力学方法分析大模型注意力谱,提升幻觉检测精度。
Thermodynamic Signatures of Reasoning: Free-Energy and Spectral-Form-Factor Diagnostics for Hallucination Detection in Large Language Models

- 将注意力图拉普拉斯算子视为哈密顿量,提取自由能等热力学量作为谱特征
- 在6个模型、6个数据集上,检测平均比基线高6.5个AUROC点
- 无需训练即可检测幻觉,适合部署场景下的无监督异常识别
大型语言模型的幻觉检测对实际部署至关重要。已有研究表明,注意力导出的图拉普拉斯谱蕴含推理质量信号。但以往谱诊断仅使用少数特征值或人工选取标量,浪费了大部分结构信息。本文提出自由能特征(Fes),将每一层注意力拉普拉斯算子视为哈密顿量,提取其配分函数、自由能、谱熵、热容以及随机矩阵理论(RMT)的谱形因子。理论证明:(i)Fes在注意力扰动下具有利普希茨稳定性;(ii)Fes可逼近由矩导出的谱泛函,在正则性和网格分辨率假设下具备表达能力;(iii)基于Fes构建的无训练探测器在有限样本下具有可靠的PAC边界。实验显示,在六个开源大模型和六个基准测试中,轻量级探测器在整体平均上达到最优,较LapEig提升+6.5 AUROC,较GoR-4提升+2.4,且无需更新模型。在完全无监督设置下,基于RMT偏差的得分实现平均AUROC 0.71,提供无标签检测能力。互补分析表明,正确生成呈现更多威格纳-迪森类谱统计,而幻觉则更接近泊松类。
原文摘要 · Abstract (English)
Hallucination detection in large language models (LLMs) is deployment-critical, and recent work shows that the spectrum of attention-derived graph Laplacians carries strong signal about reasoning quality. Prior spectral diagnostics, however, summarize the Laplacian spectrum by a handful of eigenvalues or hand-picked scalars, leaving most of its structure unused. We propose Free-Energy Signatures (Fes), a spectral descriptor that treats each layer's attention Laplacian as a Hamiltonian and extracts its thermodynamic potentials partition function, free energy, spectral entropy, heat capacity together with the random-matrix-theory (RMT) spectral form factor. We prove three results: (i)~Lipschitz stability of Fes under attention perturbation; (ii)~an expressiveness result showing that Fes enriches finite spectral summaries and approximates moment-derived spectral functionals under explicit regularity and grid-resolution assumptions; and (iii)~a finite-sample PAC bound on the AUROC of a training-free detector built from Fes. Empirically, across six open-weight LLMs and six benchmarks, a lightweight probe on Fes descriptors achieves the strongest aggregate AUROC among attention-spectral baselines, improving over LapEig by $+6.5$ AUROC points and over GoR-4 by $+2.4$ points on average, while requiring no update to the underlying LLM. In the fully unsupervised setting, an RMT-deviation score achieves mean AUROC $0.71$, providing a label-free but weaker detector. A complementary RMT analysis shows that correct generations exhibit more Wigner-Dyson like spectral statistics, whereas hallucinations exhibit more Poisson-like statistics. The anonymized code and config are provided in the supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。