揭秘隐式思维模型如何一步步推理，发现其推理路径依赖任务长度。
Do Latent-CoT Models Think Step-by-Step? A Mechanistic Study on Sequential Reasoning Tasks
- 通过分析隐藏状态，发现模型在短任务中完整生成中间步骤。
- 长任务中仅保留后期中间状态，且最终答案由最后输入与中间态融合得出。
- 揭示模型会因任务难度变化而采用捷径策略，提示设计鲁棒目标的重要性。
隐式思维链（Latent-CoT）旨在不输出长推理过程的情况下实现逐步计算，但其机制尚不明确。本文针对连续思维教师-学生蒸馏模型CODI，在严格顺序的多项式迭代任务上进行研究。通过梯度光镜解码、线性探针、注意力分析和激活修补等方法，定位中间状态表示并追踪其至最终输出的路径。在两跳和三跳任务中，CODI形成完整的桥梁状态集，这些状态可在不同隐藏思维位置被解码，而初始输入则走独立的近直接路径；预测结果通过思考末尾边界的晚期融合产生。对于更长的跳跃长度，CODI无法可靠执行完整的隐式推演，而是表现出部分隐式推理路径，集中于后期中间状态，并与最后一个输入在答案读出位置融合。消融实验表明，该部分路径在任务难度提升时可能崩溃。总体而言，本文厘清了CODI类隐式思维链在何时能实现忠实的迭代计算，何时转向压缩或捷径策略，并指出了设计稳健隐式思维链目标的挑战。
原文摘要 · Abstract (English)
Latent Chain-of-Thought (Latent-CoT) aims to enable step-by-step computation without emitting long rationales, yet its mechanisms remain unclear. We study CODI, a continuous-thought teacher-student distillation model, on strictly sequential polynomial-iteration tasks. Using logit-lens decoding, linear probes, attention analysis, and activation patching, we localize intermediate-state representations and trace their routing to the final readout. On two- and three-hop tasks, CODI forms the full set of bridge states that become decodable across latent-thought positions, while the final input follows a separate near-direct route; predictions arise via late fusion at the end-of-thought boundary. For longer hop lengths, CODI does not reliably execute a full latent rollout, instead exhibiting a partial latent reasoning path that concentrates on late intermediates and fuses them with the last input at the answer readout position. Ablations show that this partial pathway can collapse under regime shifts, including harder optimization. Overall, we delineate when CODI-style latent-CoT yields faithful iterative computation versus compressed or shortcut strategies, and highlight challenges in designing robust latent-CoT objectives for sequential reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。