arXiv:2509.07339cs.AI2025-09被引 15

发现思维链长度与问题难度无关,可能只是模型记忆的产物。

Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity

  • 用A*算法生成迷宫数据训练模型,精确测量问题复杂度。
  • 模型在简单任务上常生成冗长错误的推理链,且长度与真实解法无关。
  • 相关性仅出现在贴近训练分布的问题,说明是记忆而非真正思考。

中间标记生成(ITG)被提出用于提升语言模型在推理任务上的表现,其推理链(即思维链,CoT)与性能提升存在相关性,但机制仍不明确。社区普遍将较长的推理链视为‘思考’的证据,认为其反映问题适应性计算。本文通过在A*搜索算法的推导轨迹上从头训练变压器模型,以操作次数精确衡量迷宫问题的复杂度。实验发现,即使在最简单的自由空间问题上,模型也常生成过长的推理链,甚至无法生成正确解。在分布外问题上进一步评估,发现中间标记长度与真实A*轨迹长度仅呈松散相关。相关性仅出现在接近训练分布的问题中,表明其源于近似回忆而非真正的问题适应性计算。结果挑战了中间序列生成可反映问题复杂度的假设,警示不应将长序列自动解读为‘思考努力’。

原文摘要 · Abstract (English)

Intermediate token generation (ITG), where a model produces output before the solution, has been proposed as a method to improve the performance of language models on reasoning tasks. While these reasoning traces or Chain of Thoughts (CoTs) are correlated with performance gains, the mechanisms underlying them remain unclear. A prevailing assumption in the community has been to anthropomorphize these tokens as "thinking", treating longer traces as evidence of higher problem-adaptive computation. In this work, we critically examine whether intermediate token sequence length reflects or correlates with problem difficulty. To do so, we train transformer models from scratch on derivational traces of the A* search algorithm, where the number of operations required to solve a maze problem provides a precise and verifiable measure of problem complexity. We first evaluate the models on trivial free-space problems, finding that even for the simplest tasks, they often produce excessively long reasoning traces and sometimes fail to generate a solution. We then systematically evaluate the model on out-of-distribution problems and find that the intermediate token length and ground truth A* trace length only loosely correlate. We notice that the few cases where correlation appears are those where the problems are closer to the training distribution, suggesting that the effect arises from approximate recall rather than genuine problem-adaptive computation. This suggests that the inherent computational complexity of the problem instance is not a significant factor, but rather its distributional distance from the training data. These results challenge the assumption that intermediate trace generation is adaptive to problem difficulty and caution against interpreting longer sequences in systems like R1 as automatically indicative of "thinking effort".

思维链模型推理可信度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。