让大模型在冲突知识下更听外部信息，不依赖微调
ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- 通过轻量激活控制，引导模型优先使用外部上下文
- 在ConFiQA上显著提升忠实度，优于多个基线方法
- 无需微调、低开销，适合各类大模型快速部署
大型语言模型在预训练中存储了大量参数化知识。随着世界知识更新，其有效应用越来越依赖于忠实遵循外部检索到的上下文。当外部证据与模型内部知识冲突时，模型常默认采用记忆中的事实，导致输出不忠实。本文提出ContextFocus，一种轻量级激活调控方法，在知识冲突场景下提升上下文忠实性，同时保持流畅性和效率。该方法无需模型微调，推理开销极小。我们在ConFiQA基准上评估，对比了ContextDPO、COIECD及基于提示的方法。结果表明，ContextFocus显著提升上下文忠实性，且与提示策略互补，在更大模型上仍有效。实验验证了其有效性、鲁棒性与高效性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) encode vast amounts of parametric knowledge during pre-training. As world knowledge evolves, effective deployment increasingly depends on their ability to faithfully follow externally retrieved context. When such evidence conflicts with the model's internal knowledge, LLMs often default to memorized facts, producing unfaithful outputs. In this work, we introduce ContextFocus, a lightweight activation steering approach that improves context faithfulness in such knowledge-conflict settings while preserving fluency and efficiency. Unlike prior approaches, our solution requires no model finetuning and incurs minimal inference-time overhead, making it highly efficient. We evaluate ContextFocus on the ConFiQA benchmark, comparing it against strong baselines including ContextDPO, COIECD, and prompting-based methods. Furthermore, we show that our method is complementary to prompting strategies and remains effective on larger models. Extensive experiments show that ContextFocus significantly improves contextual-faithfulness. Our results highlight the effectiveness, robustness, and efficiency of ContextFocus in improving contextual-faithfulness of LLM outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。