用生成概率时间序列检测大模型幻觉,高效且无需访问内部参数。
HALT: Hallucination Assessment via Log-probs as Time series
- 基于输出概率构建时间序列,用门控循环单元捕捉模型校准偏差。
- 在十大能力任务上比现有方法快60倍,模型体积小30倍仍更准确。
- 适合需要快速部署、不开放内部参数的场景,如企业级私有模型。
幻觉仍是大语言模型在安全关键领域的主要障碍。本文提出HALT(基于对数概率的时间序列幻觉评估),一种轻量级幻觉检测器,仅利用LLM生成时前20个词元的对数概率作为时间序列进行分析。HALT结合门控循环单元与基于熵的特征,学习模型校准偏差,提供一种极高效的替代方案,无需大型编码器。与白盒方法不同,它不需访问隐藏状态或注意力图;与黑盒方法不同,其处理对象是概率而非表面文本,因此具备更强的领域泛化能力,并可兼容闭源模型而无需内部权重。为评估性能,我们引入HUB(幻觉检测统一基准),整合已有数据集形成十项能力,涵盖算法、常识、数学、符号、代码生成等推理任务,以及对话、数据转文本、问答、摘要、世界知识等通用技能。尽管模型尺寸仅为Lettuce的1/30,HALT在HUB上仍表现更优,且推理速度提升60倍。HALT与HUB共同构建了一个覆盖多样化LLM能力的高效幻觉检测框架。
原文摘要 · Abstract (English)
Hallucinations remain a major obstacle for large language models (LLMs), especially in safety-critical domains. We present HALT (Hallucination Assessment via Log-probs as Time series), a lightweight hallucination detector that leverages only the top-20 token log-probabilities from LLM generations as a time series. HALT uses a gated recurrent unit model combined with entropy-based features to learn model calibration bias, providing an extremely efficient alternative to large encoders. Unlike white-box approaches, HALT does not require access to hidden states or attention maps, relying only on output log-probabilities. Unlike black-box approaches, it operates on log-probs rather than surface-form text, which enables stronger domain generalization and compatibility with proprietary LLMs without requiring access to internal weights. To benchmark performance, we introduce HUB (Hallucination detection Unified Benchmark), which consolidates prior datasets into ten capabilities covering both reasoning tasks (Algorithmic, Commonsense, Mathematical, Symbolic, Code Generation) and general purpose skills (Chat, Data-to-Text, Question Answering, Summarization, World Knowledge). While being 30x smaller, HALT outperforms Lettuce, a fine-tuned modernBERT-base encoder, achieving a 60x speedup gain on HUB. HALT and HUB together establish an effective framework for hallucination detection across diverse LLM capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。