通过注意力矩阵的谱特征，无需训练即可判断大模型是否真正推理。
Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning
- 将注意力矩阵视为加权图，提取四类无参谱诊断指标。
- 在7个模型上实现85%~96%分类准确率，效应量最大达d=3.30。
- 适用于非正式思维链与证明搜索，适合研究推理本质的学者。
验证语言模型是真正推理还是模式匹配仍是开放问题：已有的验证器成本高，基于输出的启发式方法脆弱。我们发现有效的数学推理会在Transformer注意力中引发可测量、无需训练的谱信号。将每个注意力矩阵视为加权标记图，我们提取了四个无参诊断指标：费德勒值、高频能量比（HFER）、谱熵和光滑性。在四个架构族的七个模型上实验显示，效应量高达Cohen's $d = 3.30$（$p < 10^{-116}$），单阈值分类准确率达85%–96%。两个发现深化了理解：第一，柏拉图有效性——该谱信号追踪逻辑连贯性而非编译器接受度，因超时或缺少导入被拒的证明仍被正确分类，经人工审计确认（κ=0.82，n=51）；第二，架构决定性——滑动窗口注意力将判别特征从HFER转为光滑性（d=2.09，p<10^{-48}），表明注意力设计决定了何种谱通道编码推理质量。因果消融证实该信号对应归纳头电路。该方法泛化至非正式思维链（d=0.78，p<10^{-3}），在证明搜索中，以HFER重排序使Best-of-16 Pass@1提升+4.4%–6.6%，仅需零标签即达到全监督探针98%的AUC。谱图分析是一种原则性强、架构感知的推理验证基础工具。
原文摘要 · Abstract (English)
Verifying whether a language model is genuinely reasoning or pattern-matching remains an open problem: learned verifiers are expensive, and output-based heuristics are brittle. We show that valid mathematical reasoning induces a measurable, training-free spectral signature in transformer attention. By treating each attention matrix as a weighted token graph, we extract four diagnostics: Fiedler value, High-Frequency Energy Ratio (HFER), spectral entropy, and smoothness, that require no learned parameters. Experiments across seven models from four architectural families yield effect sizes up to Cohen's $d = 3.30$ ($p < 10^{-116}$), enabling $85$--$96\%$ single-threshold classification accuracy. Two findings sharpen the interpretation. First, \emph{Platonic validity}: the spectral signal tracks logical coherence rather than compiler acceptance, proofs rejected for timeouts or missing imports are correctly classified as valid, a distinction confirmed by a manual audit ($κ= 0.82$, $n = 51$). Second, \emph{architectural determinism}: Sliding Window Attention shifts the discriminative feature from HFER to smoothness ($d = 2.09$, $p < 10^{-48}$), showing that attention design governs which spectral channel encodes reasoning quality. Causal ablation confirms the signature traces induction-head circuits. The method generalises to informal chain-of-thought ($d = 0.78$, $p < 10^{-3}$), and in proof search, HFER reranking improves Best-of-16 Pass@1 by $+4.4$--$6.6$\%, matching $98\%$ of the AUC of fully supervised probes with zero labels. Spectral graph analysis is a principled, architecture-aware primitive for reasoning verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。