轻量级检测不相关问题，让RAG系统学会该闭嘴时闭嘴。
Knowing When Not to Answer: Lightweight KB-Aligned OOD Detection for Safe RAG
- 用主成分分析压缩知识库嵌入，构建低维检测空间
- 在16个领域中实现媲美大模型的异常检测效果
- 适合需要安全、快速响应的医疗等高风险场景
检索增强生成（RAG）系统正广泛应用于高风险领域，其安全性不仅取决于回答质量，更在于是否应基于知识库（KB）回应特定查询。非领域内（OOD）查询会导致密集检索返回弱相关上下文，使生成器输出流畅但无依据的内容。本文研究一种轻量级、与知识库对齐的OOD检测机制，作为RAG系统的持续防护门。方法通过主成分分析（PCA）处理KB嵌入，在由解释方差保留（EVR）或可分性驱动的t检验排序选择的低维子空间中对查询进行评分。我们在包括新冠和物质滥用在内的16个领域上评估几何语义搜索规则与轻量分类器，并通过大模型生成攻击及真实世界4chan攻击进行压力测试。结果表明，低维检测器在性能上与提示式大模型判别器相当，同时更快、更低成本且更具可解释性。人类与大模型评估显示，OOD查询主要降低RAG输出的相关性，凸显高效外部OOD检测对维持安全、合规行为的必要性。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems are increasingly deployed in high-stakes domains, where safety depends not only on how a system answers, but also on whether a query should be answered given a knowledge base (KB). Out-of-domain (OOD) queries can cause dense retrieval to surface weakly related context and lead the generator to produce fluent but unjustified responses. We study lightweight, KB-aligned OOD detection as an always-on gate for RAG systems. Our approach applies PCA to KB embeddings and scores queries in a compact subspace selected either by explained-variance retention (EVR) or by a separability-driven t-test ranking. We evaluate geometric semantic-search rules and lightweight classifiers across 16 domains, including high-stakes COVID-19 and Substance Use KBs, and stress-test robustness using both LLM-generated attacks and an in-the-wild 4chan attack. We find that low-dimensional detectors achieve competitive OOD performance while being faster, cheaper, and more interpretable than prompted LLM-based judges. Finally, human and LLM-based evaluations show that OOD queries primarily degrade the relevance of RAG outputs, showing the need for efficient external OOD detection to maintain safe, in-scope behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。