发现大模型推理时会依赖记忆检索而非真实思考,提出新方法抑制检索捷径。
Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models
- 通过误导性线索和损坏答案测试,发现模型同时依赖推理与记忆检索
- 不同任务、模型规模和微调方式下,两种机制主导程度差异显著
- 提出FARL框架,用记忆遗忘+强化学习提升真实推理能力,适合研究模型可信性者
大型推理模型(LRMs)在复杂问题求解中展现出前所未有的链式思维(CoT)能力。然而,近期研究发现其最终答案常与其自身推理过程矛盾。我们假设这种不一致源于两个竞争机制:CoT推理与记忆检索。为验证此假设,我们在推理阶段引入误导性线索、在检索阶段使用损坏答案,对多个模型和数据集进行受控实验。结果表明,两种机制同时存在,其相对主导性受问题领域、模型规模及微调方法(如强化学习与知识蒸馏)影响。研究揭示当前推理微调范式的重大缺陷:模型可利用检索机制作为捷径,从而‘劫持’奖励信号,削弱真实推理能力的培养。为此,我们提出FARL——一种融合记忆遗忘与强化学习的新型微调框架。通过在微调过程中有意识地抑制检索捷径,FARL促进以推理为主导的行为,增强模型的泛化推理能力。代码已开源:https://github.com/ZJUWYH/FARL。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) exhibit unprecedented capabilities in solving complex problems through Chain-of-Thought (CoT) reasoning. However, recent studies reveal that their final answers often contradict their own reasoning traces. We hypothesize that this inconsistency stems from two competing mechanisms for generating answers: CoT reasoning and memory retrieval. To test this hypothesis, we conduct controlled experiments that challenge LRMs with misleading cues during reasoning and/or corrupted answers during retrieval. Our results across models and datasets confirm that both mechanisms operate simultaneously, with their relative dominance influenced by multiple factors: problem domains, model scales, and fine-tuning approaches (e.g., reinforcement learning vs. distillation). The findings reveal a critical limitation in current reasoning fine-tuning paradigms: models can exploit the retrieval mechanism as a shortcut, effectively "hacking" the reward signal and undermining genuine reasoning development. To address this challenge, we introduce FARL, a novel fine-tuning framework that integrates memory unlearning with reinforcement learning. By carefully suppressing retrieval shortcuts during the fine-tuning process, FARL promotes reasoning-dominant behavior and enhances generalizable reasoning capabilities. The code is available: https://github.com/ZJUWYH/FARL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。