通过分块因子化提升稀疏自编码器的效率与可解释性
Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders
- 将潜在空间分解为头和低维前潜变量，通过可微与门组合生成后潜特征
- 在相同计算量下性能媲美强基线，同时降低编码器计算开销
- 适合关注模型可解释性与高效压缩的开发者
稀疏自编码器（SAEs）将语言模型激活分解为稀疏且可解释的特征，但标准编码器通常将潜在词典视为独立坐标，层次结构与特征交互仅隐式出现。本文提出KronSAE，将潜在空间因子化为多个头，通过mAND（一种可微的与门式交互）将低维前潜变量组合成后潜特征，引入组合性共激活先验，同时兼容标准SAE目标及TopK、Matryoshka、Switch SAE等变体。KronSAE在EV-FLOPs前沿表现优异，提升了潜在变量可解释性，更好捕捉特征相关结构，并额外降低编码器计算成本。代码已开源：https://github.com/corl-team/kronsae。
原文摘要 · Abstract (English)
Sparse Autoencoders (SAEs) decompose language-model activations into sparse, interpretable features, but standard encoders usually treat the latent dictionary as a flat set of independent coordinates, leaving hierarchy and feature interactions to emerge only implicitly. We propose KronSAE, a design that factorizes the latent space into heads and forms post-latent features as pairwise compositions of lower-dimensional pre-latents using mAND, a differentiable AND-like interaction. This imposes a compositional co-activation prior while remaining compatible with standard SAE objectives and variants such as TopK, Matryoshka, and Switch SAEs. KronSAE matches strong baselines on the EV-FLOPs frontier, improves interpretability of the latents, better captures the underlying correlated feature structure, and reduces encoder computational cost as an additional benefit. Code is available at https://github.com/corl-team/kronsae.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。