让大模型自己识别胡说八道的内容,提升生成可靠性。
Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models
- 用输出词元的不确定性,聚合隐藏状态预测回答可信度。
- 误导性上下文会让模型自信地说错话,但系统能识别这种错误。
- 适用于需要可靠生成的对话与智能体场景。
大语言模型容易生成看似流畅却错误的内容(即幻觉),在多轮或智能体应用中尤其危险。本文研究上下文信息如何影响模型行为,并探究模型能否识别自身不可靠的回答。提出一种基于分词级别不确定性的可靠性估计方法,通过输出逻辑值计算认知和随机不确定性,识别关键词元并聚合其隐藏状态,生成用于回答层面可靠性预测的紧凑表示。在开放问答基准上的控制实验表明,正确上下文可同时提高准确率与模型置信度,而误导性上下文常导致高自信错误回答,暴露了不确定性与正确性之间的不一致。基于探针的方法成功捕捉模型行为变化,在多个开源LLM上提升了不可靠输出的检测能力。结果揭示直接不确定性信号的局限性,并强调基于不确定性的探针在可靠性感知生成中的潜力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are prone to generating fluent but incorrect content, known as confabulation, which poses increasing risks in multi-turn or agentic applications where outputs may be reused as context. In this work, we investigate how in-context information influences model behavior and whether LLMs can identify their unreliable responses. We propose a reliability estimation that leverages token-level uncertainty to guide the aggregation of internal model representations. Specifically, we compute aleatoric and epistemic uncertainty from output logits to identify salient tokens and aggregate their hidden states into compact representations for response-level reliability prediction. Through controlled experiments on open QA benchmarks, we find that correct in-context information improves both answer accuracy and model confidence, while misleading context often induces confidently incorrect responses, revealing a misalignment between uncertainty and correctness. Our probing-based method captures these shifts in model behavior and improves the detection of unreliable outputs across multiple open-source LLMs. These results underscore the limitations of direct uncertainty signals and highlight the potential of uncertainty-guided probing for reliability-aware generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。