arXiv:2605.28149cs.LG2026-05

让单个神经元同时表示相反语义,提升语言模型解释效率

Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations

论文配图:Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations
图 1 · 摘自论文原文
  • 设计双向跳跃激活函数,让一个神经元承载正负两种语义方向
  • 在六个测试点上,半宽模型性能超越全宽基线,死神经元减少35%~82%
  • 适合需要精准因果控制的模型可解释性研究者使用

稀疏自编码器(SAEs)从大语言模型激活中提取可解释特征,但传统方法强制潜变量非负,导致相反语义(如“压力过高”与“压力过低”)需拆分到两个潜变量,浪费字典容量。本文提出符号感知门控稀疏自编码器(SA-GSAE),结合双侧门控稀疏性、带符号的无收缩幅值和辅助门控监督,引入新型Bi-Jump-ReLU激活函数,使单个潜变量可同时表示同一解码方向的正负极性;参数分析表明,即使反相关对稀少,符号感知仍保持参数高效。在Pythia-1B和SmolLM3-3B的三个中等深度钩子点(六组单元,三组种子)上,半宽SA-GSAE在六组中的三组上优于全宽门控SAE的平均前沿,在其余三组中R²差距小于0.025,且在匹配L_0=64时死神经元比例降低0.35–0.82。消融实验表明双侧门控和辅助损失至关重要,而单极不对称性不必要;推荐采用完全对称的绑定变体作为默认配置。盲审语义审计发现,所有测试基线及SA-GSAE中,单个潜变量两极间命名对立现象罕见;符号条件干预显示,单个有符号潜变量可作为双向因果旋钮,而一对非负潜变量则无法实现类似功能,因此解释范围应相应限定。在全宽配置下,SA-GSAE过度参数化,其报告配置在SmolLM3-3B残差流位置出现可复现的重构崩溃;建议配置(小阈值初始化 + 死潜变量阈值重置)可有效避免。

原文摘要 · Abstract (English)

Sparse Autoencoders (SAEs) extract interpretable features from Large Language Model activations, but standard variants enforce non-negative latents, so a bidirectional semantic axis (e.g., "pressure too high" vs. "pressure too low") must be split across two latents, wasting dictionary capacity on anticorrelated features. We propose the Sign-Aware Gated SAE (SA-GSAE), which combines two-sided gated sparsity, signed shrinkage-free magnitudes, and auxiliary gate supervision in a new Bi-Jump-ReLU activation, so that a single latent carries both polarities of one decoder direction; parameter accounting shows sign-awareness stays parameter-efficient even when anticorrelated pairs are rare. Across three mid-depth hookpoints on Pythia-1B and SmolLM3-3B (six cells, three seeds), a half-width SA-GSAE empirically dominates the aggregate mean frontier of a full-width Gated SAE on three of six cells, matches its R^2 within 0.025 on the remaining three, and cuts dead fraction by 0.35-0.82 absolute at matched L_0 = 64 on all six. Ablations show the two-sided gate and the auxiliary loss are essential whereas per-polarity asymmetry is not; we recommend the fully tied symmetric variant as the default. A blinded semantic audit finds nameable opposition between a latent's two sides is rare for SA-GSAE and all tested baselines, while sign-conditioned interventions show a single signed latent acts as a bidirectional causal dial where a pair of "opposite" non-negative latents does not; we scope interpretability claims accordingly. At full width, SA-GSAE is over-parameterized and its reported configuration exhibits a reproducible reconstruction collapse at the SmolLM3-3B residual-stream site; the recommended configuration (small threshold initialization with dead-latent threshold resets) prevents it.

可解释性稀疏编码大模型分析神经元建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。