通过因果建模揭示隐式思维链的动态机制,发现其非均匀分阶段运作。
Dynamics Within Latent Chain-of-Thought: An Empirical Study of Causal Structure
- 将隐式思维链建模为结构因果模型,用干预分析其步骤因果关系。
- 早期输出已偏移但表征未承诺,存在解码与表征的滞后差距。
- 适用于想理解或改进隐式推理模型的研究者,尤其关注可解释性。
隐式或连续思维链方法用内部潜变量步骤替代显式文本推理过程,但这些中间计算难以通过相关性探针之外的方式评估。本文将潜变量思维链视为表示空间中可操控的因果过程,通过将潜变量步骤建模为结构因果模型(SCM)中的变量,并采用逐步干预(do-intervention)分析其影响。我们在数学和通用推理任务上研究了两种代表性范式(Coconut 和 CODI),回答三个核心问题:(1)哪些步骤对正确性是因果必要的?答案能否提前解码?(2)信息如何跨步骤传播?该结构与显式思维链有何差异?(3)中间轨迹是否保留竞争性答案模式?输出承诺与表征承诺有何不同?结果表明,潜变量预算的行为更像具有非局部路由的阶段性功能,而非同质扩展深度;我们还发现早期输出偏差与晚期表征承诺之间存在持续差距。这些发现推动了基于模式条件与稳定性感知的分析方法及相应训练/解码目标的发展,作为更可靠的工具以解释和优化隐式推理系统。代码已开源:https://github.com/J1mL1/causal-latent-cot。
原文摘要 · Abstract (English)
Latent or continuous chain-of-thought methods replace explicit textual rationales with a number of internal latent steps, but these intermediate computations are difficult to evaluate beyond correlation-based probes. In this paper, we view latent chain-of-thought as a manipulable causal process in representation space by modeling latent steps as variables in a structural causal model (SCM) and analyzing their effects through step-wise do-interventions. We study two representative paradigms (i.e., Coconut and CODI) on both mathematical and general reasoning tasks to investigate three key questions: (1) which steps are causally necessary for correctness and when answers become decodable early; (2) how influence propagates across steps and how this structure compares to explicit CoT; and (3) whether intermediate trajectories retain competing answer modes and how output-level commitment differs from representational commitment across steps. We find that latent-step budgets behave less like homogeneous extra depth and more like staged functionality with non-local routing, and we identify a persistent gap between early output bias and late representational commitment. These results motivate mode-conditional and stability-aware analyses, together with corresponding training/decoding objectives, as more reliable tools for interpreting and improving latent reasoning systems. Code is available at https://github.com/J1mL1/causal-latent-cot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。