通过轨迹几何分析,揭示大模型推理难易的内在规律。
Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

- 将推理过程视为隐藏空间中的曲线,用谱特征量化复杂度。
- 有效维度d_ρ达0.93 AUC,可区分题目难易;前20%生成即能预测正确性。
- 方法适用于数学推理,可指导早期终止策略,适合模型可解释性研究者。
链式思维(CoT)推理使大语言模型通过生成中间步骤解决复杂问题。尽管对推理长度和内容已有较多研究,但其内部几何结构仍不明确。本文研究了变压器模型隐藏状态空间中CoT轨迹的几何特性,将每个推理链形式化为ℝᵈ中的离散曲线,并通过谱、位置与运动学几何函数进行刻画。提出有效维度d_ρ作为轨迹复杂度的度量,理论表明特征值谱越平坦的轨迹对应更难任务,因其探索更多隐藏维度。实验上,在MATH500数据集的数学推理任务中,d_ρ在区分难易题时达到0.93 AUC;运动学特征(如均位置、平均速度、速度方差等)可在仅生成前20%标记符时预测解题正确性。这些正确性信号在不同难度问题间具有迁移性,证明模型内部推理轨迹形状是任务难易与解质量的原理性窗口。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the \emph{geometry} of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in $\mathbb{R}^d$ and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension $d_ρ$ as a measure of trajectory complexity and show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on mathematical reasoning problems from the MATH500 dataset, $d_ρ$ achieves $0.93$ AUC in distinguishing easy from hard problems, while kinematic features potentially can predict correctness from only the first $20\%$ of generated tokens. These correctness signatures transfer across questions of varying difficulty, establishing that the shape of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。