用开关标记让隐藏推理可训练可分析,提升模型决策透明度。
Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learning

- 引入显式开关标记,使隐状态推理兼容在线强化学习。
- 在相同规模下性能优于已有隐状态推理方法。
- 可直接探测和干预推理过程,揭示关键计算节点位置。
隐状态链式思维通过连续隐状态替代显式推理路径来压缩推理过程,但现有方法难以用标准在线强化学习(RL)优化,也缺乏因果可解释性。本文提出SWITCH框架,通过一对显式边界标记<swi>和</swi>同时解决上述问题:离散标记使GRPO策略比值在每个决策点均有定义,支持标准在线强化学习;同一标记也为机制分析提供自然切入点。模型通过可见到隐式推理的渐进式训练,并采用可穿过循环隐计算传播梯度的Switch-GRPO目标函数。实验显示SWITCH在相似规模下持续优于先前方法。机制分析进一步发现:(i) <swi>是局部聚焦、可学习的切换策略而非风格化伪影;(ii) 其开启的隐层步骤执行问题相关且因果重要的计算;(iii) 该计算集中于进入时的单个隐状态转移。结果表明,隐状态递归推理既可被强化学习训练,也可从内部进行直接机制分析。
原文摘要 · Abstract (English)
Latent chain-of-thought compresses reasoning by replacing visible reasoning traces with continuous hidden-state recurrence, but existing formulations are difficult to optimize with standard on-policy reinforcement learning (RL) and hard to interpret causally. Our key insight is that a single pair of explicit boundary tokens can address both issues at once: discrete entry and exit anchors make the latent block compatible with standard on-policy RL, and the same anchors offer a natural foothold for mechanistic analysis. Motivated by this, we propose SWITCH, a switchable latent reasoning framework. The model emits <swi> to enter latent mode and </swi> to exit. Because the boundaries are ordinary discrete tokens, the GRPO policy ratio is well-defined at every decision point. The same anchors also expose the latent steps to direct probing and causal intervention. We train the model with a visible-to-latent curriculum and a Switch-GRPO objective that propagates gradients through recurrent latent computation. SWITCH consistently outperforms prior hidden-state-recurrence latent reasoning approaches at similar scale. Mechanistic analysis through the boundary tokens further reveals three findings: (i) <swi> is a sharply localised, learned switching policy rather than a stylistic artefact; (ii) the latent step it opens performs problem-specific, causally important computation rather than acting as an inert placeholder; and (iii) that computation is concentrated at a single hidden-state transition on entry. Together, these results show that hidden-state-recurrence latent reasoning is both RL-trainable and open to direct mechanistic analysis, including of how on-policy RL itself improves the model from the inside.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。