解释性高不等于好用,发现解释性与可控性关联弱
Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoders
- 通过90个稀疏自编码器对比分析解释性与控制效果
- 解释性与控制效果相关性仅0.298,提升效果仅52.52%
- 提出新筛选标准,让解释性真正服务于模型控制
稀疏自编码器(SAEs)被广泛用于控制大语言模型(LLMs),基于其特征可解释的假设。然而,解释性是否真的带来更好控制效果仍不明朗。我们训练了90个SAE,覆盖三个大模型(Gemma-2-2B、Qwen-2.5-3B、Gemma-2-9B),五种架构和六种稀疏度,并分别使用SAEBench和AxBench评估其解释性与控制效用,通过Kendall's tau b系数进行排名一致性分析。结果显示二者仅存在较弱正相关(tau b ≈ 0.298),表明解释性不能充分代表控制性能。我们推测原因在于特征选择偏差:并非所有可解释特征都等效用于控制。为此,提出新标准Delta Token Confidence,衡量增强某一特征对下一个词分布的影响程度。实验显示,该方法使三模型控制效果提升52.52%,相比当前最优输出分数标准。更关键的是,使用该标准筛选后,解释性与控制效用的相关性降至零(tau b ≈ 0),甚至转为负相关,凸显最有效控制特征与解释性之间的根本分歧。
原文摘要 · Abstract (English)
Sparse Autoencoders (SAEs) are widely used to steer large language models (LLMs), based on the assumption that their interpretable features naturally enable effective model behavior steering. Yet, a fundamental question remains unanswered: does higher interpretability indeed imply better steering utility? To answer this question, we train 90 SAEs across three LLMs (Gemma-2-2B, Qwen-2.5-3B, Gemma-2-9B), spanning five architectures and six sparsity levels, and evaluate their interpretability and steering utility based on SAEBench (arXiv:2501.12345) and AxBench (arXiv:2502.23456) respectively, and perform a rank-agreement analysis via Kendall's rank coefficients (tau b). Our analysis reveals only a relatively weak positive association (tau b approx 0.298), indicating that interpretability is an insufficient proxy for steering performance. We conjecture the interpretability utility gap may stem from the selection of SAE features, as not all of them are equally effective for steering. To further find features that truly steer the behavior of LLMs, we propose a novel selection criterion called Delta Token Confidence, which measures how much amplifying a feature changes the next token distribution. We show that our method improves the steering performance of three LLMs by 52.52 percent compared to the current best output score based criterion (arXiv:2503.34567). Strikingly, after selecting features with high Delta Token Confidence, the correlation between interpretability and utility vanishes (tau b approx 0), and can even become negative. This further highlights the divergence between interpretability and utility for the most effective steering features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。