通过能量模型重解分类器，无须训练即可检测大模型幻觉。
Spilled Energy in Large Language Models
- 将LLM softmax视为能量模型，追踪生成过程中的能量溢出。
- 能量溢出与事实错误、偏见和失败高度相关，准确率超85%。
- 无需训练或激活消融，适用于预训练和指令微调模型。
我们将大型语言模型（LLM）的最终softmax分类器重新解释为能量基准模型（EBM），在推理过程中将序列到序列的概率链分解为多个相互作用的EBM。这一严谨方法使我们能够追踪解码过程中的“能量溢出”，实证显示其与事实错误、偏见和失败密切相关。与Orgad等人（2025）类似，本方法可精确定位正确答案词元并检测幻觉，但关键优势在于无需训练探测分类器或激活消融。相反，我们提出两种完全免训练的度量：能量溢出，用于捕捉连续生成步骤间理论上应一致的能量值差异；边际能量，可在单步测量。在九个基准测试上评估了包括LLaMA、Mistral和Gemma在内的前沿LLM，以及合成代数运算（Qwen3），结果表明该方法具有稳健的幻觉检测能力与跨任务泛化性能。值得注意的是，这些效果在预训练和指令微调版本中均成立，且不引入任何训练开销。代码已公开于github.com/OmnAI-Lab/spilled-energy。
原文摘要 · Abstract (English)
We reinterpret the final Large Language Model (LLM) softmax classifier as an Energy-Based Model (EBM), decomposing the sequence-to-sequence probability chain into multiple interacting EBMs at inference. This principled approach allows us to track "energy spills" during decoding, which we empirically show correlate with factual errors, biases, and failures. Similar to Orgad et al. (2025), our method localizes the exact answer token and subsequently tests for hallucinations. Crucially, however, we achieve this without requiring trained probe classifiers or activation ablations. Instead, we introduce two completely training-free metrics derived directly from output logits: spilled energy, which captures the discrepancy between energy values across consecutive generation steps that should theoretically match, and marginalized energy, which is measurable at a single step. Evaluated on nine benchmarks across state-of-the-art LLMs (including LLaMA, Mistral, and Gemma) and on synthetic algebraic operations (Qwen3), our approach demonstrates robust, competitive hallucination detection and cross-task generalization. Notably, these results hold for both pretrained and instruction-tuned variants without introducing any training overhead. Code available at: github.com/OmnAI-Lab/spilled-energy
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。