提出新方法检测大模型幻觉,通过语义一致性分析识别错误回答。
Prompt-Response Semantic Divergence Metrics for Faithfulness Hallucination and Misalignment Detection in Large Language Models
- 用提示词的多种同义改写测试回答一致性,更敏感发现幻觉
- 高分值表示严重语义偏离,能识别自信但错误的编造内容
- 适合评估大模型可靠性,尤其关注生成安全性的研究者
大型语言模型(LLMs)在生成非事实性、无意义或不忠实文本时存在幻觉问题。本文提出语义偏离度量(SDM),一种轻量级框架,用于检测忠实性幻觉——即模型输出与输入上下文严重偏离的情况。聚焦于‘编造’类错误,即回答与用户查询语义不一致且任意生成的内容。现有方法如语义熵通过单一固定提示下多答案多样性来测试任意性,而本方法更具提示感知能力:不仅考察多个回答的一致性,还考察多个语义等价的提示改写下的响应一致性。方法上,利用句子嵌入联合聚类构建提示与回答共享的主题空间,通过主题共现热力图量化用户-机器对话的二维语义关联。进而计算一系列信息论指标衡量提示与回答间的语义偏离。实用评分$\\(mathcal{S}_H$结合Jensen-Shannon散度与Wasserstein距离,分数越高表明忠实性幻觉越严重。此外,发现KL散度KL(回答||提示)是区分不同生成行为的关键信号——语义探索强度。这些指标整合为语义盒诊断框架,可分类模型输出类型,包括危险且自信的编造回应。
原文摘要 · Abstract (English)
The proliferation of Large Language Models (LLMs) is challenged by hallucinations, critical failure modes where models generate non-factual, nonsensical or unfaithful text. This paper introduces Semantic Divergence Metrics (SDM), a novel lightweight framework for detecting Faithfulness Hallucinations -- events of severe deviations of LLMs responses from input contexts. We focus on a specific implementation of these LLM errors, {confabulations, defined as responses that are arbitrary and semantically misaligned with the user's query. Existing methods like Semantic Entropy test for arbitrariness by measuring the diversity of answers to a single, fixed prompt. Our SDM framework improves upon this by being more prompt-aware: we test for a deeper form of arbitrariness by measuring response consistency not only across multiple answers but also across multiple, semantically-equivalent paraphrases of the original prompt. Methodologically, our approach uses joint clustering on sentence embeddings to create a shared topic space for prompts and answers. A heatmap of topic co-occurances between prompts and responses can be viewed as a quantified two-dimensional visualization of the user-machine dialogue. We then compute a suite of information-theoretic metrics to measure the semantic divergence between prompts and responses. Our practical score, $\mathcal{S}_H$, combines the Jensen-Shannon divergence and Wasserstein distance to quantify this divergence, with a high score indicating a Faithfulness hallucination. Furthermore, we identify the KL divergence KL(Answer $||$ Prompt) as a powerful indicator of \textbf{Semantic Exploration}, a key signal for distinguishing different generative behaviors. These metrics are further combined into the Semantic Box, a diagnostic framework for classifying LLM response types, including the dangerous, confident confabulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。