用思维链引导知识编辑,让大模型更好处理关联事实更新。
RIPPLECOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning
- 设计带思维链的演示模板,分解多跳推理逻辑
- 在MQuAKE-cf上提升准确率7.8%至87.1%,最高达33.8%以上
- 适合需要精准维护知识链的场景,如问答系统
知识编辑中的涟漪效应是大型语言模型面临的重要挑战:单个事实修改后,模型难以准确更新相关事实序列,尤其在涉及多跳推理的问题上表现不佳。现有方法转向更灵活的上下文学习(ICL)编辑,但传统ICL仅用`Imagine that + 新事实`示范,无法明确复杂多跳问题中的事实链条。记忆型编辑虽可存储所有修改,却需持续更新且开销大。为此,我们提出RippleCOT,一种融合思维链(COT)推理的新型ICL编辑方法。其演示结构为`新事实, 问题, 思考, 答案`,通过思考部分识别并拆解问题中的多跳逻辑,有效引导模型处理复杂事实链。大量实验表明,RippleCOT显著优于当前最佳方法,在MQuAKE-cf基准上准确率提升7.8%至87.1%,最高突破33.8%。
原文摘要 · Abstract (English)
The ripple effect poses a significant challenge in knowledge editing for large language models. Namely, when a single fact is edited, the model struggles to accurately update the related facts in a sequence, which is evaluated by multi-hop questions linked to a chain of related facts. Recent strategies have moved away from traditional parameter updates to more flexible, less computation-intensive methods, proven to be more effective in addressing the ripple effect. In-context learning (ICL) editing uses a simple demonstration `Imagine that + new fact` to guide LLMs, but struggles with complex multi-hop questions as the new fact alone fails to specify the chain of facts involved in such scenarios. Besides, memory-based editing maintains additional storage for all edits and related facts, requiring continuous updates to stay effective. As a result of these design limitations, the challenge remains, with the highest accuracy being only 33.8% on the MQuAKE-cf benchmarks for Vicuna-7B. To address this, we propose RippleCOT, a novel ICL editing approach integrating Chain-of-Thought (COT) reasoning. RippleCOT structures demonstrations as `newfact, question, thought, answer`, incorporating a thought component to identify and decompose the multi-hop logic within questions. This approach effectively guides the model through complex multi-hop questions with chains of related facts. Comprehensive experiments demonstrate that RippleCOT significantly outperforms the state-of-the-art on the ripple effect, achieving accuracy gains ranging from 7.8% to 87.1%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。