arXiv:2609.01936cs.CLcs.AI2026-09

提出新型分解方法,让模型注意力机制分析摆脱训练数据依赖。

Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens

论文配图:Sparse Readout Prism: Explaining Logit-Lens Scores in Features Instead of Tokens
图 1 · 摘自论文原文
  • 用稀疏读出特征分解读出矩阵,替代传统词元映射。
  • 重构对数差值效果比基线高8.9-17.3个百分点。
  • 适合研究模型内部表示与读出结构的独立关系。

语言模型的下一个词预测随层演进,光镜方法通过将中间隐藏状态解码为词元来追踪这一过程。但光镜读取结果同时受隐藏状态和读出(反嵌入矩阵)影响。我们发现,仅因训练语料不同,两个光镜对相同隐藏状态可能输出不同词元,称为语料依赖性。为独立于训练语料分析读出结构,本文提出稀疏读出棱镜(SRP),仅基于读出权重进行分解,将任意词元对数或对数差表示为稀疏读出特征的贡献之和。这揭示了读出特征作为新的分析单元,能暴露词元身份掩盖的结构,并支持跨词元、上下文、层和光镜的比较。用SRP稀疏近似替换原读出,重构的对数差值比六个基于读出行间几何关系的基线多出8.9–17.3个百分点。移除特征时,对数差值变化与对应的SRP贡献成比例。尽管词元读取随语料变化,主导读出特征仍保持稳定。由于SRP构建不依赖语料,为光镜分析提供了独立于训练语料的控制基准。

原文摘要 · Abstract (English)

A language model's prediction of its next token develops across layers, and lens methods track this process by decoding intermediate hidden states into tokens. But a lens reading reflects both the hidden state and the readout (the unembedding matrix) used to decode it. Many lenses are fit on a corpus, and we show that two lenses differing only in their fitting corpus can report different tokens for the same hidden states. We call this dependence corpus conditionality. To examine readout structure independently of the fitting corpus, we introduce Sparse Readout Prism (SRP), which decomposes the readout using only its weights and expresses any token logit or logit difference as a sum of contributions from sparse readout features. This reveals readout features as a new unit of analysis for lens readings, exposing structure that token identities can obscure and enabling comparisons across tokens, contexts, layers, and lenses. Replacing the original readout with SRP's sparse approximation reconstructs 8.9-17.3 percentage points more of the tested logit differences than the strongest of six baselines built on geometric relations among readout rows. Ablating features shifts logit differences in proportion to their SRP contributions. Although token readings vary with the fitting corpus, the dominant readout feature remains stable. Because SRP uses no corpus in its construction, it provides a control independent of the fitting corpus for lens analyses.

模型解释读出结构光镜方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。