首个专为土耳其语RAG设计的幻觉检测模型,提升低资源语言AI可靠性。
Turk-LettuceDetect: A Hallucination Detection Models for Turkish RAG Applications
- 将幻觉检测转为词级别分类,微调三种土耳其语专用编码器。
- 最优模型在测试集上F1达0.7266,结构化任务表现更优。
- 支持8192长上下文,适合实时部署,助力可信AI应用。
大型语言模型(LLMs)常产生看似合理却事实错误的信息,即幻觉。尽管检索增强生成(RAG)系统通过外部知识来约束输出,但对形态复杂、资源匮乏的语言如土耳其语而言,幻觉仍是重大挑战。本文提出Turk-LettuceDetect,首个专为土耳其语RAG应用设计的幻觉检测模型套件。基于LettuceDetect框架,将幻觉检测建模为词级别分类任务,并在包含17,790个样本的机器翻译版RAGTruth基准数据集上,微调了三种编码器:土耳其语专用ModernBERT、TurkEmbed4STS和多语言EuroBERT。实验表明,ModernBERT模型在完整测试集上取得0.7266的F1分数,尤其在结构化任务中表现突出。模型保持计算高效,支持最长8,192个标记的上下文,适用于实时部署。对比分析显示,当前顶级LLM虽召回率高,但因过度生成幻觉内容导致精度低,凸显专用检测机制的必要性。通过发布模型与翻译数据集,本工作填补了多语言NLP中的关键空白,为土耳其语及其他语言的可靠可信AI应用奠定基础。
原文摘要 · Abstract (English)
The widespread adoption of Large Language Models (LLMs) has been hindered by their tendency to hallucinate, generating plausible but factually incorrect information. While Retrieval-Augmented Generation (RAG) systems attempt to address this issue by grounding responses in external knowledge, hallucination remains a persistent challenge, particularly for morphologically complex, low-resource languages like Turkish. This paper introduces Turk-LettuceDetect, the first suite of hallucination detection models specifically designed for Turkish RAG applications. Building on the LettuceDetect framework, we formulate hallucination detection as a token-level classification task and fine-tune three distinct encoder architectures: a Turkish-specific ModernBERT, TurkEmbed4STS, and multilingual EuroBERT. These models were trained on a machine-translated version of the RAGTruth benchmark dataset containing 17,790 instances across question answering, data-to-text generation, and summarization tasks. Our experimental results show that the ModernBERT-based model achieves an F1-score of 0.7266 on the complete test set, with particularly strong performance on structured tasks. The models maintain computational efficiency while supporting long contexts up to 8,192 tokens, making them suitable for real-time deployment. Comparative analysis reveals that while state-of-the-art LLMs demonstrate high recall, they suffer from low precision due to over-generation of hallucinated content, underscoring the necessity of specialized detection mechanisms. By releasing our models and translated dataset, this work addresses a critical gap in multilingual NLP and establishes a foundation for developing more reliable and trustworthy AI applications for Turkish and other languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。