强化学习让模型更抗干扰,而非依赖中间推理。
The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoning
- 用强化学习训练国际象棋模型,提升合法走法成功率至61%。
- 删除中间推理向量后模型仍正常运行,证明其非关键作用。
- 结果表明推理主要影响训练过程,而非推理时使用。
隐式推理让语言模型在连续向量空间中进行中间计算,常被视为推理时的内部草稿板。但该假设在强化学习(RL)背景下尚未被直接验证:现有因果分析仅限于数学与逻辑任务,且仅比较单个检查点内的依赖程度。本文通过分阶段隐式推理课程训练国际象棋模型,再进行强化学习,发现合法走法率从48%基准线单调上升至61%,同时将将死误报完全消除。对同一模型在强化学习前后的六种因果干预实验显示:替换或噪声化推理向量不影响性能,移除推理向量仅导致轻微退化,仅当推理向量被精确置零时才会崩溃。关键差异在于:在精确置零下,强化学习前合法率降至1%,而强化学习后为9%,这一差距在所有测试条件下均显著存在。一个10倍更大的复现实验进一步确认:无论是否恢复序列长度,移除推理向量仍具统计显著性;替换与噪声仍与基线无异。强化学习并未增强对推理内容的依赖,而是提升了对干扰的鲁棒性。这挑战了学界默认的‘隐式推理作为推理时可访问草稿板’的观点,表明其核心作用是训练期间塑造模型参数。本文还验证了强化学习在国际象棋中的实际增益,尽管多个团队报告相同‘隐式推理+强化学习’方法未能超越监督微调(SFT)精度。
原文摘要 · Abstract (English)
Latent, or silent, reasoning lets language models carry out intermediate computation in continuous vector space instead of words, and is widely assumed to function as an internal scratchpad the model consults during inference. Whether that assumption survives reinforcement learning has not been tested directly: existing causal analyses of latent reasoning are confined to math and logic tasks, comparing reliance on thoughts within one checkpoint, never before and after RL. We train a chess-playing model through a staged latent-reasoning curriculum followed by reinforcement learning, and find legality climbs monotonically to 61% (from a 48% pre-RL baseline) while checkmate confabulation is eliminated entirely. To locate this gain, we run a six-condition causal intervention suite on the same model before and after RL: substituting or noising the thought vectors leaves performance unchanged, ablating them costs only mild degradation, and only exact-zero vectors cause collapse. This robustness gap is itself the finding: under exact-zero corruption, legality collapses to 1% pre-RL versus 9% post-RL, a gap that survives correction across the full battery. A 10x-larger replication of the post-RL checkpoint's own battery confirms this: removing the thoughts, with or without restoring sequence length, also reaches significance; substitution and noise remain indistinguishable from baseline. RL appears to add robustness to disruption, not reliance on thought content. These results push back against the field's default assumption that latent thoughts function as an actively consulted inference-time scratchpad, and instead indicate latent reasoning's principal effect here is shaping the model's parameters during training. We also demonstrate a working RL gain in chess, where multiple groups report the same latent-reasoning-plus-RL recipe failing to improve accuracy over SFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。