用推理时的稀疏自编码器动态调整生成，节省训练开销且效果不减。
DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignment
- 推理阶段动态调节稀疏自编码器激活，仅修改相关隐变量。
- 在少样本偏好数据下仍表现稳定，对齐计算量减少4.47倍。
- 适合追求高效对齐、关注可解释性的模型优化研究者。
偏好对齐通常通过在偏好数据上进行权重更新训练实现，这增加了大量对齐阶段的计算开销，且机制透明度有限。我们提出动态 SAE 驱动的偏好对齐方法（DSPA），一种推理时的方法，使稀疏自编码器（SAE）的调控具有提示条件性。基于偏好三元组，DSPA 计算出连接提示特征与生成控制特征的条件差异图；解码过程中，仅修改激活的词元隐变量，不更新基础模型权重。在 Gemma-2-2B/9B 与 Qwen3-8B 上，DSPA 提升了 MT-Bench 得分，在 AlpacaEval 上表现竞争力，同时保持多项选择准确率。在受限偏好数据条件下，DSPA 依然稳健，可媲美两阶段的 RAHF-SCIT 流水线，且对齐阶段所需浮点运算量最多减少 4.47 倍。最后，我们审计了 DSPA 修改的 SAE 特征，发现偏好方向主要由话语和风格信号主导，并提供了理论分析，阐明条件差异图估计的有效性及何时对 top-k 特征截断是合理的。
原文摘要 · Abstract (English)
Preference alignment is usually achieved by weight-updating training on preference data, which adds substantial alignment-stage compute and provides limited mechanistic visibility. We propose Dynamic SAE Steering for Preference Alignment (DSPA), an inference-time method that makes sparse autoencoder (SAE) steering prompt-conditional. From preference triples, DSPA computes a conditional-difference map linking prompt features to generation-control features; during decoding, it modifies only token-active latents, without base-model weight updates. Across Gemma-2-2B/9B and Qwen3-8B, DSPA improves MT-Bench and is competitive on AlpacaEval while preserving multiple-choice accuracy. Under restricted preference data, DSPA remains robust and can rival the two-stage RAHF-SCIT pipeline while requiring up to $4.47\times$ fewer alignment-stage FLOPs. Finally, we audit the SAE features DSPA modifies, finding that preference directions are dominated by discourse and stylistic signals, and provide theory clarifying the conditional-difference map estimate and when top-$k$ ablation is principled.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。