解决隐式推理中潜在状态趋同问题,提升逻辑多样性。
Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning

- 引入光谱正则化器WIDER,通过惩罚共线方向增强潜在空间多样性。
- 实验显示有效秩提高,主导方向能量下降,推理冗余减少。
- 适用于追求高效连续推理的模型优化,无需修改原有结构。
链式思维(CoT)通过引入中间计算步骤提升大语言模型的推理能力,但显式推理会增加解码长度、延迟和上下文开销。隐式CoT将中间推理过程嵌入连续潜在状态,更具效率。然而,潜在推理存在不稳定性:连续潜在状态可能趋于高度相似,向同一主导方向坍缩,降低推理轨迹多样性。本文提出‘潜在秩坍缩’现象,并设计轻量级谱正则化方法WIDER。训练时,WIDER估计每条潜在轨迹的共享方向,并惩罚其投影,促使潜在状态覆盖更广的表示子空间。该方法可即插即用,不改变主干模型、潜在调度及推理解码流程。进一步将坍缩建模为隐式推理中的几何瓶颈,将其缓解视为训练阶段的正则化问题而非推理阶段调整。大量实验表明,WIDER优于基线模型;机制分析揭示有效秩更高、主导方向能量更低、潜在步骤间冗余减少。结果强调潜在子空间利用对高效连续推理的重要性,为隐式CoT提供几何分析视角。代码已开源:https://github.com/whitesweater/WIDER。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning improves the reasoning ability of large language models by introducing intermediate computation, but explicit rationales increase decoding length, latency, and context cost. Implicit CoT offers a more efficient alternative by moving intermediate reasoning into continuous latent states. However, latent reasoning can be unstable: successive latent states may become overly similar and collapse toward a shared dominant direction, reducing the diversity of the reasoning trajectory. In this work, we identify $\textit{latent rank collapse}$ and propose $\textbf{WIDER}$, a lightweight spectral regularizer for implicit CoT. During training, WIDER estimates the shared direction of each latent trajectory and penalizes projections onto this direction, encouraging latent states to span a broader representational subspace. The method is plug-and-play and leaves the backbone model, latent schedule, and inference-time decoding procedure unchanged. We further formulate this collapse as a geometric bottleneck in implicit reasoning, casting its mitigation as a training-time regularization problem rather than an inference-time decoding change. Extensive experiments show that WIDER improves matched implicit CoT baselines, while mechanistic analyses reveal higher effective rank, lower dominant-direction energy, and reduced redundancy among latent steps. These results highlight latent subspace utilization as an important factor for efficient continuous reasoning, providing a geometric perspective for analyzing and improving implicit CoT. Code is available at https://github.com/whitesweater/WIDER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。