arXiv:2606.14990cs.LGcs.AI2026-06中稿 · ICML

用可学习的有理函数替代固定激活,让稀疏自编码器更灵活高效。

Rational Sparse Autoencoder

  • 用可训练的有理函数代替传统固定激活,提升模型适应性。
  • 在多个语言模型上均优于基线,重建与下游任务性能全面提升。
  • 仅增加少量参数,单卡几分钟即可完成训练,适合快速部署。

稀疏自编码器(SAE)是机制可解释性的重要工具,但现有SAE家族受限于ReLU、JumpReLU和TopK等固定编码器非线性,硬编码了特定稀疏机制,可能扭曲重建与稀疏性的权衡。本文提出有理稀疏自编码器(RSAE),将固定编码器激活替换为可训练的有理函数。有理激活能在紧致域上统一逼近现有SAE家族的激活原语(如TopK的阈值门控),同时具备更强的函数表达能力以适应预激活几何结构。实现上采用两阶段流程:先通过合成数据上的松弛Remez交换获取有理系数并复制预训练基线权重,再联合校准尺度参数与有理系数;随后在标准稀疏正则化重建目标下微调。实验证明,在三个开源语言模型的残差流激活上,对所有基线激活族和全部测试稀疏度,RSAE在重建指标与下游行为指标上均严格优于基线,且不损失特征级可解释性。性能提升在不同模型、激活族和稀疏度下均一致,升级仅增加每编码器数个标量参数,单卡几分钟即可完成。

原文摘要 · Abstract (English)

Sparse autoencoders (SAEs) are standard tools for mechanistic interpretability, but current SAE families are constrained by fixed encoder nonlinearities such as ReLU, JumpReLU, and TopK. This hard-codes a particular sparsity mechanism into the model and can distort the reconstruction-versus-sparsity trade-off. We introduce the Rational Sparse Autoencoder (RSAE), which replaces the fixed encoder activation with a trainable rational function. Rational activations are flexible enough to uniformly approximate the activation primitives used by existing SAE families on compact domains (for TopK, the thresholded gate obtained after a separating top-k threshold is supplied), while also providing a richer function class for adapting to the observed pre-activation geometry. We realise this idea through a two-stage pipeline: an initialisation procedure that copies the pre-trained baseline SAE weights, plugs in rational coefficients obtained by the relaxed Remez exchange on synthetic data, and calibrates the scale parameters along with the rational coefficients; followed by a fine-tuning step under the standard sparsity-regularised reconstruction objective. Empirically, on residual-stream activations of three open-weight language models and across all three baseline activation families, the RSAE strictly improves on it after the fine-tuning step, both on reconstruction-side metrics and on downstream-behaviour metrics, without sacrificing feature-level interpretability under sparse probing. These gains are consistent across host language models, across baseline activation families, and across the full range of baseline sparsity we tested, while the upgrade itself adds only a handful of scalar parameters per autoencoder and runs in minutes on a single consumer GPU.

稀疏自编码器可解释性有理函数模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。