动态调整解码策略,让大模型在记忆冲突时更准,无冲突时更快。
Mitigating Context-Memory Conflicts in LLMs through Dynamic Cognitive Reconciliation Decoding

- 通过注意力图分析预测记忆与上下文冲突,决定走哪条解码路径。
- 在6个数据集上优于所有基线,大模型准确率提升最高达4.2%。
- 适合需要高可靠性的问答、内容生成场景,尤其知识频繁更新的领域。
大型语言模型通过预训练积累了大量参数化知识,但当过时或错误的参数化知识与上下文中的外部知识冲突时,会影响输出质量。现有方法多采用对比解码缓解冲突,但在无冲突场景下,静态方法会破坏输出分布;其他动态方法虽尝试衡量冲突程度,仍难以应对复杂真实场景。本文提出两阶段解码方法——动态认知调和解码(DCRD),先分析注意力图评估上下文保真度并预测潜在冲突,再根据预测结果选择贪婪解码或基于上下文保真度的动态解码路径。该设计使DCRD在存在冲突时高效缓解问题,在无冲突时保持高精度与解码效率。为模拟频繁知识更新场景,我们构建了冲突知识问答基准ConflictKG。在四个LLM和六个QA数据集上的实验表明,DCRD显著优于所有基线,达到当前最优性能。
原文摘要 · Abstract (English)
Large language models accumulate extensive parametric knowledge through pre-training. However, knowledge conflicts occur when outdated or incorrect parametric knowledge conflicts with external knowledge in the context. Existing methods address knowledge conflicts through contrastive decoding, but in conflict-free scenarios, static approaches disrupt output distribution. Other dynamic decoding methods attempt to measure the degree of conflict but still struggle with complex real-world situations. In this paper, we propose a two-stage decoding method called Dynamic Cognitive Reconciliation Decoding (DCRD), to predict and mitigate context-memory conflicts. DCRD first analyzes the attention map to assess context fidelity and predict potential conflicts. Based on this prediction, the input is directed to one of two decoding paths: (1) greedy decoding, or (2) context fidelity-based dynamic decoding. This design enables DCRD to handle conflicts efficiently while maintaining high accuracy and decoding efficiency in conflict-free cases. Additionally, to simulate scenarios with frequent knowledge updates, we constructed ConflictKG, a knowledge conflict QA benchmark. Experiments on four LLMs across six QA datasets show that DCRD outperforms all baselines, achieving state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。