提出新方法量化大模型推理每一步的计算能量，发现错误推理在关键步骤能量更低。
How Hard Does It Think? Analyzing Step-Aware Reasoning Energy in LLM Chain-of-Thought Trajectories
- 用隐藏状态的核矩阵对齐度衡量每步推理的计算能耗
- 不同推理步骤能耗差异显著，错误路径在关键节点能量偏低
- 适合研究模型内部推理机制或提升推理可靠性的人参考
理解计算资源在链式思维（CoT）各步骤间的分配仍是开放挑战：现有可解释性方法依赖输出层面信号或将处理深度压缩为单一轨迹标量，导致步骤级努力不透明。我们提出步骤感知推理能量（SARE），一种基于中心核对齐（CKA）的几何框架，通过相邻Transformer层间标记隐藏状态的格拉姆矩阵对比，以细粒度捕捉个体推理步骤的计算能量，无需特征向量对齐或聚类对应。SARE进一步通过建模推理轨迹在潜在语义状态间的转移，将能量置于语义演进背景中。在六个推理基准和三个开源大模型上，我们发现推理能量在步骤类型间高度非均匀，表现出轨迹级指标无法察觉的相变式特征；错误轨迹在关键推理节点上系统性呈现更低能量；且基于SARE的特征在多数场景下表现媲美或优于基于输出置信度的基线，表明内部几何动态编码了超越表层信号的预测信息。
原文摘要 · Abstract (English)
Understanding how computational effort is allocated across individual chain-of-thought (CoT) reasoning steps remains an open challenge: existing interpretability methods rely on output-level signals or collapse processing depth into a single trajectory-level scalar, leaving step-wise effort opaque. We propose Step-Aware Reasoning Energy (SARE), a geometric framework that quantifies effort at the granularity of individual CoT steps via Centered Kernel Alignment (CKA) between Gram matrices of token hidden states across adjacent transformer layers, capturing inter-token relational structure without requiring eigenvector alignment or cluster correspondence. SARE further contextualizes this energy within reasoning's semantic progression by modeling CoT trajectories as transitions among latent semantic states. Across six reasoning benchmarks and three open-weight LLMs, we find that reasoning energy is highly non-uniform across step types, exhibiting phase-like transitions invisible to trajectory-level metrics; incorrect trajectories show systematically lower energy at critical reasoning junctions; and SARE-based features match or outperform output-based confidence baselines in most settings, indicating that internal geometric dynamics encode predictive information beyond surface-level signals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。