用编码器特征直接控制模型行为，更高效公平。
Rethinking Sparse Autoencoders: Select-and-Project for Fairness and Control from Encoder Features Alone
- 仅通过编码器选出关键特征并投影到原空间，无需重训练
- 在视觉语言模型上公平性提升最高3.2倍，大模型攻击性降低3.6倍
- 适合需要快速、无损干预模型行为的场景
稀疏自编码器（SAEs）广泛用于模型可解释性和行为控制。传统方法通过修改解码器中间表示实现控制，本质是将原始激活重写为解码器特征的加权和。本文提出一种编码器中心的替代方案，显著提升跨模态性能。我们引入S&P Top-K框架，无需重训练且计算轻量，能识别与敏感属性或行为对齐的编码器前K个特征，可选地聚合为单一控制轴，并计算正交投影直接应用于模型原嵌入空间。在视觉-语言模型中，该方法使CelebA和FairFace上的公平性指标最高提升3.2倍；在大语言模型中，显著降低Llama-3 8B Instruct的攻击性和迎合性，较掩码重建方法提升达3.6倍。结果表明，编码器中心干预比传统解码器中心使用SAEs更具通用性、效率和有效性。
原文摘要 · Abstract (English)
Sparse Autoencoders (SAEs) are widely employed for mechanistic interpretability and model steering. Within this context, steering is by design performed by means of decoding altered SAE intermediate representations. This procedure essentially rewrites the original activations as a weighted sum of decoder features. In contrast to existing literature, we forward an encoder-centric alternative to model steering which demonstrates a stronger cross-modal performance. We introduce S&P Top-K, a retraining-free and computationally lightweight Selection and Projection framework that identifies Top-K encoder features aligned with a sensitive attribute or behavior, optionally aggregates them into a single control axis, and computes an orthogonal projection to be subsequently applied directly in the model's native embedding space. In vision-language models, it improves fairness metrics on CelebA and FairFace by up to 3.2 times over conventional SAE usage, and in large language models, it substantially reduces aggressiveness and sycophancy in Llama-3 8B Instruct, achieving up to 3.6 times gains over masked reconstruction. These findings suggest that encoder-centric interventions provide a general, efficient, and more effective mechanism for shaping model behavior at inference time than the traditional decoder-centric use of SAEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。