用稀疏特征调控大模型生成语言，零样本下准确率高达90%
Causal Language Control in Multilingual Transformers via Sparse Feature Steering
- 通过修改单个稀疏编码器特征实现语言控制
- 在五种语言间切换成功率达90%，语义保持不变
- 适合需要轻量可控多语言生成的开发者
确定性控制大型多语言语言模型（LLM）的目标生成语言仍是根本性挑战，尤其在无提示、无微调的零样本设置下。本文研究是否可利用稀疏自动编码器（SAE）特征——此前被证明与可解释模型行为相关——来推理时引导LLM的生成语言。基于Gemma-2B和Gemma-9B残差流上的预训练SAE，我们识别出在英语与中文、日语、西班牙语、法语之间激活差异最大的特征。仅修改一个Transformer层中的单一SAE特征，即可实现高达90%的准确率（以FastText分类为准），同时保持语义一致性（通过LaBSE句向量相似度衡量）。分析表明，语言调控在中到晚期层最有效，并受特定注意力头放大，这些头与语言敏感的SAE特征高度关联。结果证明稀疏特征调控是一种轻量且可解释的可控多语言生成机制。
原文摘要 · Abstract (English)
Deterministically controlling the target generation language of large multilingual language models (LLMs) remains a fundamental challenge, particularly in zero-shot settings where neither explicit language prompts nor fine-tuning are available. In this work, we investigate whether sparse autoencoder (SAE) features, previously shown to correlate with interpretable model behaviors, can be leveraged to steer the generated language of LLMs during inference. Leveraging pretrained SAEs on the residual streams of Gemma-2B and Gemma-9B, we identify features whose activations differ most significantly between English and four target languages: Chinese, Japanese, Spanish, and French. By modifying just a single SAE feature at one transformer layer, we achieve controlled language shifts with up to 90\% success, as measured by FastText language classification, while preserving semantic fidelity according to LaBSE (Language-Agnostic BERT Sentence Embedding) similarity. Our analysis reveals that language steering is most effective in mid-to-late transformer layers and is amplified by specific attention heads disproportionately associated with language-sensitive SAE features. These results demonstrate the promise of sparse feature steering as a lightweight and interpretable mechanism for controllable multilingual generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。