用注意力图谱分析模型进化中的隐藏缺陷，发现语法压力下连接性崩溃现象。
Spectral Archaeology: The Causal Topology of Model Evolution
- 通过计算注意力图谱的代数连通性等指标，无须训练即可探测模型内部机制。
- 发现代码到对话转换时出现英语语法触发的连接性骤降，降幅达Δλ₂≈-0.76。
- 该现象可被简单规则识别，适合用于模型训练阶段的可解释性审计。
行为基准只能告诉我们模型做什么，却无法揭示其如何工作。我们提出一种无需训练的机制探测方法，利用注意力图谱的谱特性。将每一层视为一个词元图，计算代数连通性（λ₂）、平滑度与谱熵。在12个模型和10种语言中，这些指标生成稳定的“谱指纹”，揭示了传统评估未察觉的突变。研究发现：（1）经历特定课程过渡（如代码转对话）的模型在非标准句式上出现英语特异性连接性失效，Δλ₂≈-0.76，称此为“被动触发连接性崩溃”（PTCC）；Phi系列分析表明，该现象随发展阶段出现与恢复，指向脆弱课程设计而非合成数据本身。（2）PTCC反映专业化代价：形式路由增强，风格灵活性下降。（3）识别出四种重复出现的处理策略，仅用简单冻结阈值规则即可跨模型线准确识别。（4）机制上，PTCC集中于第2层稀疏的“补偿头”，在语法压力下失效；激活引导可部分恢复连接性，恢复约38%的信息流。此外，主导拓扑格局更受分词密度影响，而非语言身份，表明“健康”几何结构随书写系统系统性变化。整体而言，注意力图谱谱特征为模型审计与训练制度验证提供了实用工具。
原文摘要 · Abstract (English)
Behavioral benchmarks tell us \textit{what} a model does, but not \textit{how}. We introduce a training-free mechanistic probe using attention-graph spectra. Treating each layer as a token graph, we compute algebraic connectivity ($λ_2$), smoothness, and spectral entropy. Across 12 models and 10 languages, these measures yield stable ``spectral fingerprints'' that expose discontinuities missed by standard evaluation. We report four results. (1) Models undergoing specific curriculum transitions (e.g., code-to-chat) show an English-only, syntax-triggered connectivity failure on non-canonical constructions, reaching $Δλ_2 \approx -0.76$. We term this scar \textit{Passive-Triggered Connectivity Collapse} (PTCC). Analysis of the Phi lineage reveals that PTCC appears and resolves across developmental stages, implicating brittle curriculum shifts rather than synthetic data per se. (2) PTCC reflects a specialization trade-off: strengthened formal routing at the expense of stylistic flexibility. (3) We identify four recurrent processing strategies; simple frozen-threshold rules enable perfect forensic identification across lineages. (4) Mechanistically, PTCC localizes to a sparse Layer 2 ``compensatory patch'' of heads that fails under syntactic stress; activation steering can partially restore connectivity, recovering $\approx 38\%$ of lost information flow. Finally, dominant topological regimes track tokenization density more than language identity, suggesting ``healthy'' geometry varies systematically across scripts. Overall, attention-graph spectra provide a practical tool for auditing and training-regime verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。