用专家路由机制让稀疏自编码器更高效，大幅降低训练成本。
Efficient Dictionary Learning with Switch Sparse Autoencoders
- 引入专家路由结构，分治处理激活向量，减少计算开销。
- 在固定算力下，重建精度与稀疏性平衡优于现有方法。
- 特征可解释性不降反升，适合模型可解释性研究者使用。
稀疏自编码器（SAEs）是将神经网络激活分解为人类可理解特征的新兴技术。然而，要识别前沿模型中的全部特征，需将SAE扩展至极宽规模，带来巨大计算挑战。本文提出开关稀疏自编码器（Switch SAEs），受稀疏专家混合模型启发，通过路由机制将激活向量分配给多个小型专家SAE，实现SAE高效扩展以覆盖更多特征。实验对比显示，Switch SAEs在给定训练算力预算下，在重建误差与稀疏性权衡曲线上实现显著帕累托改进。我们还分析了跨专家特征的几何结构，发现特征重复现象，并验证Switch SAE提取的特征与传统SAE一样具备可解释性。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) are a recent technique for decomposing neural network activations into human-interpretable features. However, in order for SAEs to identify all features represented in frontier models, it will be necessary to scale them up to very high width, posing a computational challenge. In this work, we introduce Switch Sparse Autoencoders, a novel SAE architecture aimed at reducing the compute cost of training SAEs. Inspired by sparse mixture of experts models, Switch SAEs route activation vectors between smaller "expert" SAEs, enabling SAEs to efficiently scale to many more features. We present experiments comparing Switch SAEs with other SAE architectures, and find that Switch SAEs deliver a substantial Pareto improvement in the reconstruction vs. sparsity frontier for a given fixed training compute budget. We also study the geometry of features across experts, analyze features duplicated across experts, and verify that Switch SAE features are as interpretable as features found by other SAE architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。