arXiv:2606.27321cs.LGcs.AI2026-06

给顶k稀疏自编码器加软正则化,让特征更专注可解释。

Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders

论文配图:Beyond the Hard Budget: Sparsity Regularizers for More Interpretable Top-k Sparse Autoencoders
图 1 · 摘自论文原文
  • 在选中前对激活值施加软正则,提升特征选择性
  • 两种正则均提升单义性,且不损害重建质量
  • 适合想提升模型可解释性的研究人员

稀疏自编码器(SAEs)已成为解析视觉基础模型表示的核心工具,将多义激活分解为更多稀疏、更单义的特征。顶-k SAE通过激活函数在结构上强制稀疏性,仅保留每输入中活跃度最高的k个隐变量。由于设计初衷是避免早期SAE使用的ℓ₁惩罚及其缺陷,该方法未与显式稀疏正则化结合。然而顶-k SAE仍存在局限,我们假设在选择前施加稀疏惩罚可增强每个隐变量的选择性,提升代码可解释性,同时不引入ℓ₁惩罚的弊端。本文提出两种兼容顶-k架构的稀疏正则化:对未选中(非支持集)单元施加ℓ₁惩罚,以及尺度不变的ℓ₁/ℓ₂比值惩罚,使信息集中在更少有效单元上。两者仅作用于批内至少一次被选中的活跃单元。在两个数据集、三种视觉基础模型及多个k值下,两种正则化均持续提升单义性,且不影响重建质量。ℓ₁/ℓ₂惩罚进一步压缩信息至更少隐变量,增强对推理时k值选择的鲁棒性,并提升小预算线性探测性能。核心发现是:硬性结构稀疏与软性正则化具有互补性而非互斥性。

原文摘要 · Abstract (English)

Sparse autoencoders (SAEs) have become a leading tool for interpreting the representations of vision foundation models, decomposing their polysemantic activations into a larger set of sparse, more monosemantic features. The Top-$k$ SAE, a now-standard variant, enforces sparsity architecturally through its activation function, retaining only the $k$ most active latents per input. Because it was designed precisely to avoid the $\ell_1$ penalty used by earlier SAEs and its known drawbacks, it has not been combined with an explicit sparsity regularizer. Yet the Top-$k$ SAE retains limitations of its own, and we hypothesize that a sparsity penalty acting before the selection could sharpen each latent's selectivity and make the code more interpretable, without reintroducing the drawbacks of the $\ell_1$ penalty. We introduce two sparsity regularizers compatible with the Top-$k$ architecture, both acting on the activations before the Top-$k$ selection: an $\ell_1$ penalty on the unselected (off-support) units, and a scale-invariant $\ell_1/\ell_2$-ratio penalty that concentrates the code onto fewer effective units. Both penalties are applied only to the batch-active units, those selected by the Top-$k$ operator at least once within the batch. Across two datasets, three vision foundation models, and a range of $k$, both regularizers consistently improve monosemanticity at no cost to reconstruction quality. The $\ell_1/\ell_2$ penalty further concentrates information into fewer latents, making reconstruction more robust to the inference-time choice of $k$ and improving small-budget linear probing. Our central finding is that hard architectural sparsity and soft sparsity regularization are complementary rather than mutually exclusive.

稀疏编码可解释性自编码器

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。