arXiv:2608.03291cs.LGcs.AI2026-08

通过分析推理过程动态,发现模型错误的隐藏模式并纠正。

The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics

论文配图:The Tell-Tale Trace: Detecting Reasoning Failures in LLMs Using Chain-of-Thought Dynamics
图 1 · 摘自论文原文
  • 用推理轨迹的演化规律识别错误,不依赖语义真假
  • 在布尔可满足性任务中,失败轨迹提前终止且重复操作
  • 针对错误模式干预后,准确率从13.3%提升至85%

链式思维(CoT)推理提升了大语言模型性能,并提供了可观测的推理过程接口。现有方法多关注单个中间步骤的语义正确性或一致性,而忽略了推理过程在整个轨迹中的动态演变。因此,分布于整个推理路径中的失败仍被忽视。此外,显式表述的CoT未必忠实反映模型内部推理,促使我们探索不依赖语义真实性的分析方式。本文研究多种大语言模型在可验证布尔可满足性任务上的表现,控制复杂度以逼近模型能力边界。通过标注推理步骤的功能,发现:在可满足问题中,错误轨迹会过早进入子句检查,重复相似操作并提前结束;在不可满足问题中,模型会错误地急于得出可满足结论,检查候选赋值而非推导矛盾。随后,采用针对性的证明搜索提示干预,使Llama3-70B的准确率从13.3%提升至85%,纠正了84.6%的错误。结果表明,能力缺陷可能表现为可见推理结构的分布式、任务相关变化,且不依赖语义忠实性的CoT动态分析可用于诊断和修正失败。

原文摘要 · Abstract (English)

Chain-of-thought (CoT) reasoning improves large language model (LLM) performance while also providing an observable interface to the model's reasoning process. Existing approaches that leverage verbalized CoTs to monitor reasoning correctness, however, largely evaluate the semantic correctness or consistency of individual intermediate steps, rather than how the reasoning process evolves across the trace. As a result, failures distributed across the reasoning trajectory, rather than those localized to a single incorrect step, remain comparatively underexplored. Furthermore, verbalized CoTs need not faithfully reflect the model's internal reasoning, motivating analyses that do not treat individual statements as literal accounts of internal computation. In this work, we therefore ask whether the dynamics of visible CoT can be leveraged to systematically distinguish successful from failed reasoning without assuming such semantic faithfulness. We study a range of LLMs on verifiable Boolean satisfiability tasks with variable complexity, enabling controlled comparisons near each model's capability frontier. Tagging CoT sentences by reasoning function reveals premature verification collapse on SAT problems: incorrect traces enter clause checking earlier, repeat similar operations, and finalize sooner. On UNSAT problems, models presumptuously move towards incorrect SAT conclusions, checking candidate assignments rather than deriving contradictions across constructed cases. Subsequently, a targeted proof-search prompt intervention raises Llama3-70B accuracy from 13.3% to 85%, correcting 84.6% of these errors. These results show that capability failures can manifest as distributed, task-dependent changes in the structure of visible reasoning, and that CoT dynamics agnostic to whether the verbalized trace reflects the model's internal computations can help diagnose and correct failures.

大模型推理错误检测链式思维

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。