arXiv:2608.05660cs.LGcs.CL2026-08

通过融合运动与位置信息,提升大模型推理错误检测精度。

Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs

论文配图:Reasoning Errors Have a Region and a Direction in the Residual-Stream Trajectory of LLMs
图 1 · 摘自论文原文
  • 设计三路检测器,结合运动与区域、方向的受限状态视图。
  • 在未见过的推理基准上,准确率比纯位移方法高12%。
  • 适用于推理、事实补全与验证,信号更贴近正确性而非推理类型。

随着语言模型被用于需可验证推理的任务,可靠区分有效推理与错误推理已成为关键问题。现有基于轨迹的方法依赖层间残差流位移,捕捉表示变化并抑制部分稳定、标记特异的信息。但位移忽略了更新的起始状态,而恢复完整状态又可能引入捷径依赖信息。本文识别出这一权衡,提出三路检测器:基于向量量化构建粗粒度区域读取器,以及在归一化多层状态上进行细粒度方向读取器。该设计恢复足够状态上下文以解释运动,同时避免全状态探测。在训练中未见的推理基准上,本方法相比仅使用位移的最先进方法,选择准确率提升最高达12%,相比单层探测基线提升21%。尽管仅在推理基准上训练,其在事实补全和事实验证任务上也优于所有对比检测器,表明信号聚焦于正确性而非推理形式。消融实验进一步显示,运动、区域与方向提供互补信号。结果表明,推理有效性应从状态条件下的运动中读取,而非静态状态或去上下文的轨迹本身。

原文摘要 · Abstract (English)

As language models are increasingly used for tasks that require verifiable reasoning, reliably distinguishing sound reasoning from flawed reasoning has become an important practical problem. Recent trajectory-based methods seek this signal in layerwise residual-stream displacements, which capture how representations change while attenuating some stable, token-specific information. However, displacement omits the state from which an update originates, whereas restoring the full state risks reintroducing shortcut-prone information. We identify this trade-off and propose a three-stream detector that combines motion with two restricted views of location. A coarse region reader based on vector quantization and a fine direction reader over normalized multi-layer states. This design restores enough state context to interpret the motion without returning to full-state probing. On reasoning benchmarks unseen during training, our method improves selection accuracy by up to 12% over the displacement-only state of the art and 21% over single-layer probing baselines. Although trained only on reasoning benchmarks, it also reads factual completion and fact verification, ahead of every detector we compare against, which places the signal on correctness rather than on a kind of reasoning. Ablations further show that motion, region, and direction provide complementary signals. These results suggest that reasoning validity is better read from state-conditioned motion than from either static states or decontextualized trajectories alone.

大模型推理轨迹分析错误检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。