arXiv:2505.13775cs.LGcs.AI2025-05被引 51

模型推理过程看似合理,实则可能毫无逻辑,却仍能得出正确答案。

Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens

  • 用正确和错误的中间步骤训练模型,结果表现相近
  • 错误推理步骤的模型在新任务上泛化能力更强
  • 推理长度与问题复杂度无关,不能反映真实思考

近期大型推理模型的出色表现常被归因于思维链(Chain of Thought, CoT)机制。然而,这种中间推理步骤是否真正反映模型内部计算过程尚不明确。本文通过在形式可验证的推理轨迹上从零训练变换器模型,系统研究其作用。结果显示:即使使用完全正确的中间步骤,模型仍可能生成无效推理;而使用与问题无关的错误中间步骤训练的模型,性能与正确训练组相当,甚至在分布外任务上表现更优。此外,基于GRPO的强化学习后处理虽提升解题准确率,但未改善推理有效性。最后发现,推理轨迹长度与问题实际计算复杂度基本无关。这些结果挑战了‘中间令牌代表可预测推理行为’的假设,警示不应将语言模型的输出过度拟人化或解读为类人类或算法行为的证据。

原文摘要 · Abstract (English)

Recent impressive results from large reasoning models have been interpreted as a triumph of Chain of Thought (CoT), and especially of the process of training on CoTs sampled from base LLMs in order to help find new reasoning patterns. While these traces certainly seem to help model performance, it is not clear how they influence it, with some works ascribing semantics to them and others cautioning against relying on them as transparent and faithful proxies of the model's internal computational process. To systematically investigate the role of end-user semantics of derivational traces, we set up a controlled study where we train transformer models from scratch on formally verifiable reasoning traces and the solutions they lead to. We notice that, despite gains over the solution-only baseline, models trained on entirely correct traces can still produce invalid reasoning traces even when arriving at correct solutions. More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, perform similarly to those trained on correct ones, and even generalize better on out-of-distribution tasks. We also study the effect of GRPO-based RL post-training on trace validity, noting that while solution accuracy increases, this is not accompanied by improvements in trace validity. Finally, we examine whether reasoning-trace length reflects inference-time scaling and find that trace length is largely agnostic to the underlying computational complexity of the problem being solved. These results challenge the assumption that intermediate tokens or ``Chains of Thought'' reflect or induce predictable reasoning behaviors and caution against anthropomorphizing such outputs or over-interpreting them (despite their mostly seemingly forms) as evidence of human-like or algorithmic behaviors in language models.

推理机制思维链模型可信度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。