让大模型更信外部证据,用轻量方法提升事实准确性
Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time

- 分离注意力路由与信号强度,动态增强上下文信息
- 在不改注意力分布下提升输入信号信噪比
- 无需训练,即插即用,兼顾准确与流畅
大语言模型在面对与内部参数记忆冲突的输入证据时,常出现'忽略上下文'现象,导致持续的事实幻觉。现有缓解方法多依赖抑制神经元激活或计算昂贵的对比解码,往往增加困惑度或显著提高推理延迟。为此,我们提出轻量级推理时干预方法——共振上下文锚定(RCA),基于残差流信号动力学视角,解决外部证据在深层网络传播中的信号衰减问题。核心机制是在自注意力模块中对路由逻辑与信息幅度进行正交解耦:利用原始softmax前的注意力分数作为语义对齐瞬时指标,通过非线性修正构建动态增益场,选择性放大对应上下文标记的值向量范数,而不改变注意力概率分布。该机制有效提升了残差流混合中的输入证据信噪比,从而在推理过程中稳健锚定生成轨迹至真实上下文。在Llama-3系列模型上的大量实验表明,RCA显著提升多个事实一致性与强知识冲突任务中的上下文忠实度,有效抑制参数化幻觉。结果还证实,作为无训练、计算开销可忽略的即插即用模块,RCA在忠实度与流畅度之间实现帕累托改进,同时保持模型通用语言理解能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) frequently exhibit "contextual disregard" when faced with input evidence that conflicts with their internal parametric memory, leading to persistent factual hallucinations. Existing mitigation strategies primarily rely on suppressing specific neuron activations or employing computationally expensive contrastive decoding mechanisms, which often result in increased perplexity or significantly elevated inference latency. To address these limitations, we propose Resonant Context Anchoring (RCA), a lightweight inference-time intervention method grounded in the perspective of residual stream signal dynamics. RCA aims to resolve the signal attenuation of external evidence during its propagation through deep networks. The core mechanism involves the orthogonal decoupling of routing logic and information magnitude within the self-attention module. By utilizing raw pre-softmax attention scores as an instantaneous metric of semantic alignment, we construct a dynamic gain field via non-linear rectification to selectively amplify the norms of value vectors corresponding to context tokens, without altering the attention probability distribution. This mechanism effectively elevates the signal-to-noise ratio (SNR) of input evidence within the residual stream mixture, thereby robustly anchoring the generation trajectory to the truthful context during inference. Extensive experiments on the Llama-3 model series demonstrate that RCA significantly improves contextual faithfulness across multiple factual consistency and strong knowledge-conflict tasks, effectively suppressing parametric hallucinations. Furthermore, results confirm that as a training-free and computationally negligible plug-and-play module, RCA achieves a Pareto improvement in faithfulness and fluency while maintaining the model's general language understanding capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。