发现大模型慢思考源于逆树结构冻结,通过调控训练时机提升推理能力。
Emergent Slow Thinking in LLMs as Inverse Tree Freezing
- 将模型推理建模为概念网络中的随机游走,路径合并与竞争驱动逆树形成
- 逆树结构导致推理链长度随拓扑稀疏性几何增长,标准训练在高预算下崩溃
- 在最大挫败时刻插入微调可避免灾难性遗忘,显著提升分布外性能
基于可验证奖励的强化学习(RLVR)使大语言模型从稀疏最终答案信号中习得多步慢思考。本文提出统计物理视角:自回归模型有限容量迫使指数级前缀空间压缩为预测状态的马尔可夫网络,慢思考表现为其中的随机游走——即概念网络(CoNet)图景。在该框架下,RLVR动态由两类机制主导:兼容路径合并与不兼容路径间的挫败竞争。二者协同推动网络经历成核、生长并冻结为多输入单输出的有向逆树结构。该模型复现了15亿参数大模型的训练动态,并提出三项预测:推理链长度因拓扑稀疏性呈几何增长;监督微调(SFT)通过桥接节点断裂引发灾难性遗忘;挫败驱动策略崩溃。基于逆树冻结的结构时序特性,提出‘退火式RLVR’——在最大挫败时刻短暂插入SFT。其在分布内与分布外基准上均优于标准RLVR,尤其在高采样预算下优势显著。若延迟至逆树冻结后施加相同SFT,则触发灾难性遗忘,凸显时机的关键作用。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) enables large language models to acquire slow, multi-step reasoning from sparse final-answer signals. We provide a statistical-physics picture of this emergence. We show that an autoregressive model's finite capacity forces it to compress its exponentially large prefix space into a Markov network of predictive states, on which slow thinking unfolds as a random walk -- the Concept Network (CoNet) picture. Within CoNet, RLVR dynamics are governed by two mechanisms: merging of compatible paths and frustrated competition among incompatible ones. Together they drive the network through nucleation, growth, and freezing into multi-input, single-output directed inverse trees. The picture reproduces the training dynamics of a 1.5-billion-parameter LLM and yields three predictions: reasoning chains lengthen as a geometric necessity of sparse topology; SFT induces catastrophic forgetting through bridge-node rupture; and frustration drives policy collapse. Building on the structural timing inherent in inverse-tree freezing, we propose Annealed-RLVR -- a brief SFT intervention at the moment of maximum frustration. It outperforms standard RLVR on both in- and out-of-distribution benchmarks, with the largest gains at high sampling budgets where standard RLVR collapses. The same SFT applied after the trees freeze instead triggers catastrophic forgetting, isolating timing as the active ingredient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。