推理微调让模型内部状态长期稳定,提升多步推理能力
Reasoning Fine-Tuning Induces Persistent Latent Policy States

- 将思维链建模为切换动态系统,挖掘隐藏的状态结构
- 微调后模型状态更丰富,状态转换更有序,持久性更强
- 适合研究模型推理机制或做过程级控制的学者
推理专用语言模型相比基础模型有显著性能提升,但其内部变化机制仍不明确:是提升了局部词元能力,还是全局重构了推理过程?本文将思维链建模为切换动态系统(SDS),通过时间感知对比表示学习与离散模式发现,从激活轨迹中恢复隐藏策略。在四个基准测试和1.5B至32B参数规模模型上,推理微调模型展现出比基础模型更丰富的隐状态组织,表现为状态间转换结构更分化,状态使用率、持续性和混合度具有模型依赖性变化。恢复出的模式与不同推理阶段功能对应,多种对照实验表明其结构并非由正确性、表示学习或先验假设解释,而是依赖推理轨迹的时序一致性。因果干预显示这些状态具有实际功能:状态互换会降低单步预测拟合度,将推理动态移植到基础模型可提升复杂推理任务表现。此外,基于SDS的失败前缀剪枝在12组设置中优于自洽性方法,最高提升12.5个百分点。结果表明,推理微调全局重构了隐状态动力学,为推理模型的机理分析与过程控制提供新视角。
原文摘要 · Abstract (English)
Reasoning-specialized language models show large performance gains over base models, yet the internal changes responsible for improved multi-step reasoning remain poorly understood. It is unclear whether reasoning fine-tuning improves local token-level competence or globally reorganizes how models structure inference over time. We address this question by modeling Chain-of-Thought reasoning as a switching dynamical system (SDS), in which internal representations evolve under discrete latent policy states. Our framework combines time-aware contrastive representation learning with discrete regime discovery to recover latent policies from activation trajectories. Across four benchmarks and model scales from 1.5B to 32B parameters, reasoning-fine-tuned models exhibit richer latent-policy organization than their base counterparts, characterized by more differentiated transition structure and model-dependent changes in state utilization, persistence, and mixing. The recovered regimes exhibit functional specialization aligned with distinct reasoning stages, and extensive controls confirm that their structure is not explained by correctness, representation learning, or modeling priors, but depends on the coherent temporal organization of reasoning trajectories. Causal interventions further show that the regimes are functionally meaningful: state-swap ablations reduce one-step predictive fit, while transplanting reasoning dynamics into base models improves performance on challenging reasoning problems. Finally, SDS-guided pruning of failure-prone reasoning prefixes outperforms self-consistency in 11 of 12 model-dataset settings, with gains of up to 12.5 percentage points. Together, our results suggest that reasoning fine-tuning globally reorganizes latent dynamics, offering a new lens for mechanistic analysis and process-level control of reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。