探针看似能解码时间信息,实则与模型真实计算无关,揭示了可解释性研究的潜在盲区。
When and How Long? The Readout-Mediator Angle in Temporal Reasoning

- 通过角度分析发现,探针方向与模型实际计算路径正交。
- 移除模型关键子空间会完全崩溃性能,而探针方向无关紧要。
- 适用于怀疑探针可靠性或研究模型内部机制的研究者。
线性探针几乎完美地解码了语言模型中日历日期持续时间的表示,但其方向对模型答案无影响;相比之下,通过分布式对齐搜索(DAS)发现的四维子空间被移除后,性能彻底崩溃。测量这两个子空间之间的夹角,发现其与随机子空间间夹角无异,表明探针学习的是与模型实际计算正交的方向。逆向工程显示:注意力头通过±30和±61天的查询键偏置传递月粒度上下文,多层感知机将绝对日期转换为持续时间,均位于探针未触及的因果子空间下游。稀疏自编码器分解证实:探针对齐特征与DAS对齐特征语义分离,因果重叠极小。该现象在1.5-9B参数量级、两个模型家族中重复出现,并初步扩展至空间位移与符号运算领域,暗示探针-中介正交是探针可解释性方法的普遍失效模式。这直接挑战了将探针用于运行时安全监控的设想:探针可能在高置信度报告一个模型早已放弃的方向。
原文摘要 · Abstract (English)
A linear probe can decode a representation almost perfectly and yet be completely irrelevant to how the model uses it. On calendar-date duration reasoning in language models, a $\sin$/$\cos$ probe recovers day-of-year from a layer's activations, yet ablating its direction has no effect on the model's answers -- while ablating a four-dimensional subspace found by Distributed Alignment Search (DAS) at the same layer collapses performance entirely. We measure the angle between these two subspaces -- the \emph{readout-mediator angle} -- and find it indistinguishable from the angle between two random subspaces (the Haar-uniform null), meaning the probe has learned a direction orthogonal to the model's actual computation. Reverse-engineering the circuit reveals why: attention heads route month-grained context through learned QK offsets at ${\pm}30$ and ${\pm}61$ days, and MLPs then convert \emph{when} (absolute date) into \emph{how long} (duration) -- all downstream of the causal subspace the probe never touches. Sparse-autoencoder decomposition confirms the split: probe-aligned and DAS-aligned features encode semantically disjoint concepts with negligible causal overlap. The dissociation replicates across four scales ($1.5$-$9\,$B) and two model families, with preliminary evidence on two further domains (spatial displacement, symbolic arithmetic), suggesting that readout-mediator orthogonality is a general failure mode of probe-based interpretability. This directly undermines proposals to deploy probes as runtime safety monitors: the probe can report high confidence on a direction the model has silently abandoned.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。