arXiv:2601.05679cs.LG2026-01被引 9

sparse autoencoders可能只是捕捉了推理的线索,而非真正的推理特征。

Do Sparse Autoencoders Identify Reasoning Features in Language Models?

  • 用反证法结合因果注入和LLM构造反例,检验特征可靠性
  • 45%-90%的候选特征仅需少量关键词就激活,说明敏感性强
  • 适合关注模型内部表征可解释性的研究人员

我们研究稀疏自编码器(SAEs)在大语言模型中识别推理相关内部特征的可靠性。通过理论分析发现,稀疏性正则化解码倾向于保留稳定的低维关联特征,而抑制行为内部的高维变化。基于此,我们提出一种基于反证的评估框架,结合因果标记注入与LLM引导的反例生成。在跨越多个模型族、层和推理数据集的22种配置中,许多对比选择的候选特征对标记级干预极为敏感:仅注入少数相关标记至非推理文本后,激活率高达45%-90%。对于剩余上下文依赖型候选特征,该方法能生成触发激活的非推理输入,以及保持语义但抑制激活的顶级激活推理样本的同义改写。一次小型控制实验显示,在评估基准上影响微弱。总体表明,在所研究设置下,稀疏分解更可能偏好与推理共现的低维线索,强调在将高层行为归因于单个SAE特征时必须进行反证检验。代码已公开于https://github.com/GeorgeMLP/reasoning-probing。

原文摘要 · Abstract (English)

We study how reliably sparse autoencoders (SAEs) support claims about reasoning-related internal features in large language models. We first give a stylized analysis showing that sparsity-regularized decoding can preferentially retain stable low-dimensional correlates while suppressing high-dimensional within-behavior variation, motivating the possibility that contrastively selected "reasoning" features may concentrate on cue-like structure when such cues are coupled with reasoning traces. Building on this perspective, we propose a falsification-based evaluation framework that combines causal token injection with LLM-guided counterexample construction. Across 22 configurations spanning multiple model families, layers, and reasoning datasets, we find that many contrastively selected candidates are highly sensitive to token-level interventions, with 45%-90% activating after injecting only a few associated tokens into non-reasoning text. For the remaining context-dependent candidates, LLM-guided falsification produces targeted non-reasoning inputs that trigger activation and meaning-preserving paraphrases of top-activating reasoning traces that suppress it. A small steering study yields minimal changes on the evaluated benchmarks. Overall, our results suggest that, in the settings we study, sparse decompositions can favor low-dimensional correlates that co-occur with reasoning, underscoring the need for falsification when attributing high-level behaviors to individual SAE features. Code is available at https://github.com/GeorgeMLP/reasoning-probing.

模型解释稀疏编码推理探查

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。