仅用单语数据就能找到控制大模型输出语言的稀疏特征。
LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steering
- 通过随机词过滤法从单语数据中筛选出语言特异性特征。
- 发现的语言特征极稀疏且对目标语言高度专一，可显著提升生成准确率。
- 无需平行语料，适合资源有限场景下的多语言控制应用。
大型语言模型具备强大的多语言能力，但可靠控制其输出语言仍具挑战。表示层操控通过在推理时向模型激活添加语言特异性向量来解决此问题，但识别残差流中的语言方向通常依赖昂贵的多语或平行数据。稀疏自编码器（SAEs）能将残差激活分解为可解释的稀疏特征方向，是寻找语言特异性方向的理想基础，但现有方法仍受限于数据需求。我们提出LangFIR（基于随机词过滤的语言特征识别），仅需少量单语数据与随机词序列即可发现语言特异性SAE特征。许多由目标语言输入持续激活的SAE特征并不编码语言身份；随机词序列能暴露这些语言无关特征，使LangFIR得以将其剔除，从而分离出一组稀疏的语言特异性特征。实验表明，这些特征极为稀疏、对目标语言高度选择性，且具有因果重要性：定向消融仅在对应语言上增加交叉熵损失。利用这些特征构建的操控向量，在三个模型（Gemma 3 1B、Gemma 3 4B、Llama 3.1 8B）、三个数据集和十二种目标语言下，多语言生成控制任务的平均准确率×BLEU优于所有现有方法，相比最强单语基线最高提升4.7倍，超越使用平行数据的方法。结果表明，多语言大模型中的语言身份存在于一组可通过单语数据发现的稀疏特征方向中。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) show strong multilingual capabilities, yet reliably controlling the language of their outputs remains difficult. Representation-level steering addresses this by adding language-specific vectors to model activations at inference time, but identifying language-specific directions in the residual stream often relies on multilingual or parallel data that can be expensive to obtain. Sparse autoencoders (SAEs) decompose residual activations into interpretable, sparse feature directions and offer a natural basis for this search, yet existing SAE-based approaches face the same data constraint. We introduce LangFIR (Language Feature Identification via Random-token Filtering), a method that discovers language-specific SAE features using only a small amount of monolingual data and random-token sequences. Many SAE features consistently activated by target-language inputs do not encode language identity. Random-token sequences surface these language-agnostic features, allowing LangFIR to filter them out and isolate a sparse set of language-specific features. We show that these features are extremely sparse, highly selective for their target language, and causally important: directional ablation increases cross-entropy loss only for the corresponding language. Using these features to construct steering vectors for the multilingual generation control task, LangFIR achieves the best average accuracy x BLEU among steering methods across three models (Gemma 3 1B, Gemma 3 4B, and Llama 3.1 8B), three datasets, and twelve target languages, outperforming the strongest monolingual baseline by up to 4.7x and surpassing methods that use parallel data. Our results suggest that language identity in multilingual LLMs is localized in a sparse set of feature directions discoverable with monolingual data. Code is available at https://github.com/JaMussCraft/LangFIR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。