通过逐词分析，揭示大模型推理中记忆错误的来源与影响。
Diagnosing Memorization in Chain-of-Thought Reasoning, One Token at a Time
- 基于统计共现识别每个推理词的记忆来源（局部、中程、远距离）
- 发现局部记忆是错误主因，导致高达67%的错误词汇出现
- 可预测错误发生的具体推理步骤，适用于多种结构化生成任务
大型语言模型在推理基准测试中表现良好，但输入微小变化时常失败，引发对其成功是否依赖记忆的担忧。这一问题在链式思维（CoT）推理中尤为严重，虚假记忆模式可能引发中间错误并导致最终答案错误。我们提出STIM框架，实现对推理链中每个词的记忆来源（局部、中程、远距离）的源感知逐词识别，依据其在预训练语料库中的统计共现特征。跨任务与分布设置的词级分析表明，模型在复杂或长尾场景下更依赖记忆，且局部记忆往往是错误的主要驱动因素，导致高达67%的错误词产生。此外，STIM得出的记忆分数能有效预测错误出现在推理的哪一步。该框架为诊断和改进模型推理提供了有力工具，并可推广至其他结构化分步生成任务。
原文摘要 · Abstract (English)
Large Language Models (LLMs) perform well on reasoning benchmarks but often fail when inputs alter slightly, raising concerns about the extent to which their success relies on memorization. This issue is especially acute in Chain-of-Thought (CoT) reasoning, where spurious memorized patterns can trigger intermediate errors that cascade into incorrect final answers. We introduce STIM, a novel framework for Source-aware Token-level Identification of Memorization, which attributes each token in a reasoning chain to one of multiple memorization sources - local, mid-range, or long-range - based on their statistical co-occurrence with the token in the pretraining corpus. Our token-level analysis across tasks and distributional settings reveals that models rely more on memorization in complex or long-tail cases, and that local memorization is often the dominant driver of errors, leading to up to 67% of wrong tokens. We also show that memorization scores from STIM can be effective in predicting the wrong tokens in the wrong reasoning step. STIM offers a powerful tool for diagnosing and improving model reasoning and can generalize to other structured step-wise generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。