一个稀疏自编码器可同时适配多种稀疏度，兼顾解释性与精度。
Train One Sparse Autoencoder Across Multiple Sparsity Budgets to Preserve Interpretability and Accuracy
- 设计新目标函数，单模型支持多稀疏度训练。
- 在Gemma-2 2B上实现稀疏度与解释方差的帕累托最优。
- 高稀疏度下仍保持良好可解释性，适合灵活部署。
稀疏自编码器（SAEs）通过稀疏约束将神经网络隐藏层表示分解为解耦且可解释的特征，是解释神经网络的重要工具。然而传统SAE受限于训练时固定的稀疏度，若需不同稀疏度则需训练多个模型，显著增加训练与推理开销。本文提出新型训练目标HierarchicalTopK，使单个SAE能同时优化多个稀疏度下的重建性能。在Gemma-2 2B上的实验表明，该方法在稀疏度与解释方差之间实现了帕累托最优，优于在单一稀疏度下训练的传统SAE。进一步分析显示，HierarchicalTopK在较高稀疏度下仍保持高可解释性分数。该方法有效弥合了SAE设计中灵活性与可解释性之间的关键缺口。
原文摘要 · Abstract (English)
Sparse Autoencoders (SAEs) have proven to be powerful tools for interpreting neural networks by decomposing hidden representations into disentangled, interpretable features via sparsity constraints. However, conventional SAEs are constrained by the fixed sparsity level chosen during training; meeting different sparsity requirements therefore demands separate models and increases the computational footprint during both training and evaluation. We introduce a novel training objective, \emph{HierarchicalTopK}, which trains a single SAE to optimise reconstructions across multiple sparsity levels simultaneously. Experiments with Gemma-2 2B demonstrate that our approach achieves Pareto-optimal trade-offs between sparsity and explained variance, outperforming traditional SAEs trained at individual sparsity levels. Further analysis shows that HierarchicalTopK preserves high interpretability scores even at higher sparsity. The proposed objective thus closes an important gap between flexibility and interpretability in SAE design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。