通过分析模型输出的完整概率分布,实现快速精准的幻觉与数据污染检测。
Beyond Next Token Probabilities: Learnable, Fast Detection of Hallucinations and Data Contamination on LLM Output Distributions
- 利用模型输出的完整概率分布序列作为特征,而非仅依赖单个词概率
- 在多个模型和基准上表现优于现有方法,检测延迟极低
- 轻量级网络设计,可跨模型和数据集迁移使用
大语言模型的安全部署亟需自动化检测幻觉和训练数据污染。在无法访问模型内部机制的情况下,当前方法通常仅依赖文本中实际出现词的概率,依赖简单任务特定启发式规则,忽视了完整下一词概率分布序列所蕴含的信息。本文提出超越人工设计决策规则,直接学习大语言模型可观测输出的全部内容——不仅包括下一词概率,还包括完整的下一词分布序列,称之为LLM输出签名(LOS),并将其作为检测幻觉与数据污染的参考数据类型。为此,我们引入LOS-Net,一种基于轻量注意力架构的模型,通过对LOS的高效编码进行训练,可证明性地逼近广泛现有的检测技术。实验表明,LOS-Net在多种基准和模型上均取得更优性能,同时保持极低检测延迟,并展现出良好的跨数据集与跨模型迁移能力。完整代码已公开于https://github.com/BarSGuy/Beyond-next-token-probabilities。
原文摘要 · Abstract (English)
The automated detection of hallucinations and training data contamination is pivotal to the safe deployment of Large Language Models (LLMs). These tasks are particularly challenging in settings where no access to model internals is available. Current approaches in this setup typically leverage only the probabilities of actual tokens in the text, relying on simple task-specific heuristics. Crucially, they overlook the information contained in the full sequence of next-token probability distributions. We propose to go beyond hand-crafted decision rules by learning directly from the complete observable output of LLMs -- consisting not only of next-token probabilities, but also the full sequence of next-token distributions. We refer to this as the LLM Output Signature (LOS), and treat it as a reference data type for detecting hallucinations and data contamination. To that end, we introduce LOS-Net, a lightweight attention-based architecture trained on an efficient encoding of the LOS, which can provably approximate a broad class of existing techniques for both tasks. Empirically, LOS-Net achieves superior performance across diverse benchmarks and LLMs, while maintaining extremely low detection latency. Furthermore, it demonstrates promising transfer capabilities across datasets and LLMs. Full code is available at https://github.com/BarSGuy/Beyond-next-token-probabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。