让大模型不仅答对，还要推理过程正确，解决知识编辑中的幻觉问题。
Reason-KE++: Aligning the Process, Not Just the Outcome, for Faithful LLM Knowledge Editing
- 通过分阶段奖励机制，监督推理中间步骤的准确性。
- 在MQUAKE-CF-3k上达到95.48%的准确率，比之前提升5.28%。
- 适合需要高可信度推理的场景，如医疗、法律等专业领域。
在复杂的多跳推理任务中，使大语言模型（LLMs）忠实于新知识是一项关键但尚未解决的挑战。我们发现，基于SFT的方法（如Reason-KE）虽为当前最优，却存在‘忠实性缺口’：其优化目标是格式模仿而非合理推理。这一缺口使得模型强大的参数先验会覆盖新上下文事实，导致严重事实幻觉（例如，尽管明确修改，仍错误推理出“休斯顿”来自“NASA”）。为解决此核心对齐问题，我们提出Reason-KE++，一种SFT+RL框架，旨在实现推理过程层面的忠实性。其核心是阶段感知奖励机制，对中间推理步骤（如分解、子答案正确性）提供密集监督。关键发现：单纯的结局导向强化学习是误导性的陷阱，会破坏推理完整性（如19.00%的跳跃准确率下降），尽管表面提升了最终准确率。我们的过程感知框架在MQUAKE-CF-3k上达到95.48%的新最佳表现（+5.28%），表明对于复杂任务，对齐推理过程是构建可信大模型的关键。
原文摘要 · Abstract (English)
Aligning Large Language Models (LLMs) to be faithful to new knowledge in complex, multi-hop reasoning tasks is a critical, yet unsolved, challenge. We find that SFT-based methods, e.g., Reason-KE, while state-of-the-art, suffer from a "faithfulness gap": they optimize for format mimicry rather than sound reasoning. This gap enables the LLM's powerful parametric priors to override new contextual facts, resulting in critical factual hallucinations (e.g., incorrectly reasoning "Houston" from "NASA" despite an explicit edit). To solve this core LLM alignment problem, we propose Reason-KE++, an SFT+RL framework that instills process-level faithfulness. Its core is a Stage-aware Reward mechanism that provides dense supervision for intermediate reasoning steps (e.g., Decomposition, Sub-answer Correctness). Crucially, we identify that naive outcome-only RL is a deceptive trap for LLM alignment: it collapses reasoning integrity (e.g., 19.00% Hop acc) while superficially boosting final accuracy. Our process-aware framework sets a new SOTA of 95.48% on MQUAKE-CF-3k (+5.28%), demonstrating that for complex tasks, aligning the reasoning process is essential for building trustworthy LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。