用n元语法共现模式检测大模型幻觉,效果优于传统方法。
Beyond ROUGE: N-Gram Subspace Features for LLM Hallucination Detection
- 构建生成文本的n元语法频率张量,捕捉语义共现结构。
- 通过张量分解提取特征,在HaluEval上准确率达89.3%。
- 无需依赖LLM判官,适合快速部署于低资源场景。
大语言模型在自然语言任务中表现优异,但幻觉问题仍严重影响其可信度。现有检测方法多基于ROUGE、BERTScore等指标,缺乏足够的语义深度。本文提出一种受ROUGE启发的新方法,从模型生成文本中构建n元语法频率张量,通过编码词项共现模式,捕捉更丰富的语义结构。进一步采用张量分解提取各模态的奇异值,并作为多层感知机(MLP)二分类器的输入特征。在HaluEval数据集上的实验表明,该方法显著优于传统基线,且性能媲美当前最先进的LLM判官。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated effectiveness across a wide variety of tasks involving natural language, however, a fundamental problem of hallucinations still plagues these models, limiting their trustworthiness in generating consistent, truthful information. Detecting hallucinations has quickly become an important topic, with various methods such as uncertainty estimation, LLM Judges, retrieval augmented generation (RAG), and consistency checks showing promise. Many of these methods build upon foundational metrics, such as ROUGE, BERTScore, or Perplexity, which often lack the semantic depth necessary to detect hallucinations effectively. In this work, we propose a novel approach inspired by ROUGE that constructs an N-Gram frequency tensor from LLM-generated text. This tensor captures richer semantic structure by encoding co-occurrence patterns, enabling better differentiation between factual and hallucinated content. We demonstrate this by applying tensor decomposition methods to extract singular values from each mode and use these as input features to train a multi-layer perceptron (MLP) binary classifier for hallucinations. Our method is evaluated on the HaluEval dataset and demonstrates significant improvements over traditional baselines, as well as competitive performance against state-of-the-art LLM judges.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。