后训练让大模型长出专用推理注意力头,但过度思考可能引发错误。
Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
- 通过电路分析发现后训练催生新型专用注意力头,支持结构化推理。
- 不同训练方式影响头演化:蒸馏/微调累积稳定头,GRPO动态迭代筛选头。
- 关闭推理不等于停用思考,反而激活低效补偿性注意力头。
现代大型推理模型的强大能力主要通过后训练技术(如监督微调SFT和强化学习RL)实现,但其架构机制仍不清晰。本文通过电路分析表明,复杂推理的后训练会激发新型、功能专一的注意力头,这些头共同支撑结构化推理与计算。跨模型家族的对比分析显示,不同训练策略下这些头的演化路径不同:知识蒸馏和SFT促进稳定推理头的累积叠加;而组相对策略优化(GRPO)则处于动态搜索模式——较少注意力头被迭代激活、评估并修剪,其存续紧密跟随任务奖励信号波动。此外,可控“思考开关”模型并无专属“思考”头;关闭显式推理后,反而触发更广泛但效率更低的补偿性注意力头。通过消融和定性分析,我们揭示了这些电路级动态与宏观性能间的关联:强化的注意力头虽能解决复杂问题,但也可能导致‘过度思考’失败模式,如简单任务中的计算错误或逻辑循环。这些发现将电路动态与整体表现联系起来,指出现有推理系统中复杂推理与基础计算间的内在权衡。研究进一步为未来训练策略设计提供方向,强调需在有效推理策略与可靠无误执行之间取得平衡。
原文摘要 · Abstract (English)
The remarkable capabilities of modern large reasoning models are largely unlocked through post-training techniques such as supervised fine-tuning (SFT) and reinforcement learning (RL). However, the architectural mechanisms behind such improvements remain largely opaque. In this work, we use circuit analysis to demonstrate that post-training for complex reasoning sparks the emergence of novel, functionally specialized attention heads. These heads collectively support structured reasoning and computation. Our comparative analysis across various model families reveals that these emergent heads evolve differently under different training regimes. Distillation and SFT foster a cumulative addition of stable reasoning heads. In contrast, group relative policy optimization (GRPO) operates in a dynamic search mode: relatively few attention heads are iteratively activated, evaluated, and pruned, with their survival closely tracking fluctuations in the task reward signal. Furthermore, we find that controllable "think on/off" models do not possess dedicated "thinking" heads. Instead, turning off explicit reasoning triggers a broader-but less efficient-set of compensatory heads. Through ablation and qualitative analyses, we connect these circuit-level dynamics to a crucial performance trade-off: strengthened heads enable sophisticated problem-solving strategies for difficult problems but can also introduce "over-thinking" failure modes, such as calculation errors or logical loops on simpler tasks. These findings connect circuit-level dynamics to macro-level performance, identifying an inherent tension where complex reasoning comes at the cost of elementary computations. More broadly, our work points to future directions for training policy design, emphasizing the need to balance the development of effective reasoning strategies with the assurance of reliable, flawless execution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。