让大模型学会判断外部信息可信度,避免被错误引用误导。
To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts
- 通过自评估信心和规则判断,动态调节对外部信息的信任程度。
- 在错误上下文场景下,主流模型仍过度依赖外部信息,准确率下降显著。
- GPT系列用自我推理提升可信度,小模型更适合规则驱动方法。
大型语言模型(LLMs)常通过检索增强生成(RAG)引入外部上下文,但这些信息可能不准确或被故意误导,与模型内部知识产生冲突。我们提出‘情境忠实性’概念,即模型应根据自身知识置信度与外部信息质量动态调整信任。为此,我们构建了一个基于Reddit真实错误上下文的新QA数据集,并在多个基准上评估发现:无论开源或闭源模型,在正确与错误上下文并存时均过度依赖外部信息。为此,我们提出两种方法:自我引导信心推理(SCR)和基于规则的信心推理(RCR)。SCR让模型自主评估外部信息可信度以生成最优答案,适用于强推理模型;RCR则提取模型输出中的显式置信信号,按预设规则决策。实验显示,对GPT-4o和GPT-4o mini,SCR相较直接输入增强基线最高提升24.2%;对Llama-3-8B,RCR更优。通过提出的信心推理直接偏好优化(CR-DPO)微调,Llama-3-8B在已见与未见数据集上平均提升8.9%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate or intentionally misleading, leading to conflicts with the model's internal knowledge. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context to resolve knowledge conflicts. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-assess the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。