对比了稀疏自编码器的渐进编码方法，发现嵌套结构在重建和语义相似性上更优。
Empirical Evaluation of Progressive Coding for Sparse Autoencoders
- 用子集剪枝实现渐进编码，替代联合训练多尺寸模型
- 嵌套式SAE在重建损失和语言建模损失上更低，语义相似性更高
- 剪枝后的普通SAE更易解释，但性能略逊于嵌套结构
稀疏自编码器（SAEs）通过无监督方式从神经网络中提取可解释特征，广泛应用于表征工程与信息检索。然而，其计算成本较高，尤其在需要多个不同规模的SAE时。本文发现，普通SAE中字典重要性遵循幂律分布。我们在语言建模任务中比较了基于子集剪枝的渐进编码方法与联合训练嵌套式SAE（即“套娃”SAE）的效果。结果表明，套娃SAE在重建损失、语言建模损失恢复以及表征相似性方面均表现更优；而剪枝后的普通SAE虽更易解释，但性能稍弱。本文探讨了这一权衡关系的成因及其影响。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) \citep{bricken2023monosemanticity,gao2024scalingevaluatingsparseautoencoders} rely on dictionary learning to extract interpretable features from neural networks at scale in an unsupervised manner, with applications to representation engineering and information retrieval. SAEs are, however, computationally expensive \citep{lieberum2024gemmascopeopensparse}, especially when multiple SAEs of different sizes are needed. We show that dictionary importance in vanilla SAEs follows a power law. We compare progressive coding based on subset pruning of SAEs -- to jointly training nested SAEs, or so-called {\em Matryoshka} SAEs \citep{bussmann2024learning,nabeshima2024Matryoshka} -- on a language modeling task. We show Matryoshka SAEs exhibit lower reconstruction loss and recaptured language modeling loss, as well as higher representational similarity. Pruned vanilla SAEs are more interpretable, however. We discuss the origins and implications of this trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。