LettuceDetect高效检测RAG生成中的幻觉，支持长文本且推理快。
LettuceDetect: A Hallucination Detection Framework for RAG Applications
- 基于ModernBERT的长上下文能力，用令牌分类识别幻觉
- 在RAGTruth数据集上达到79.22%的例级F1，比前人提升14.8%
- 模型仅30倍小，单卡每秒可处理30-60个样本，适合部署
检索增强生成（RAG）系统虽引入外部知识，仍易产生幻觉。本文提出LettuceDetect框架，解决现有方法两大缺陷：传统编码器模型受上下文窗口限制，大模型方法计算效率低。基于支持8k tokens的ModernBERT，并在RAGTruth基准数据集上训练，该方法超越所有先前编码器模型及多数提示型模型，体积仅为最优模型的1/30。LettuceDetect为令牌分类模型，处理上下文-问题-回答三元组，实现细粒度幻觉定位。在RAGTruth数据集上，例级检测F1达79.22%，较前驱模型Luna提升14.8%。系统单卡可每秒处理30至60个实例，显著提升实际应用可行性。
原文摘要 · Abstract (English)
Retrieval Augmented Generation (RAG) systems remain vulnerable to hallucinated answers despite incorporating external knowledge sources. We present LettuceDetect a framework that addresses two critical limitations in existing hallucination detection methods: (1) the context window constraints of traditional encoder-based methods, and (2) the computational inefficiency of LLM based approaches. Building on ModernBERT's extended context capabilities (up to 8k tokens) and trained on the RAGTruth benchmark dataset, our approach outperforms all previous encoder-based models and most prompt-based models, while being approximately 30 times smaller than the best models. LettuceDetect is a token-classification model that processes context-question-answer triples, allowing for the identification of unsupported claims at the token level. Evaluations on the RAGTruth corpus demonstrate an F1 score of 79.22% for example-level detection, which is a 14.8% improvement over Luna, the previous state-of-the-art encoder-based architecture. Additionally, the system can process 30 to 60 examples per second on a single GPU, making it more practical for real-world RAG applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。