发现跨语言翻译的因果特征,揭示模型中真正起作用的通用翻译方向。
Recurrence Is Not Enough: Causally Validating Multilingual SAE Translation Features in Gemma 2 and 3
- 通过多语言提示测试,验证稀疏自编码器特征在不同语言间的因果作用。
- 20多个重复出现的特征中仅1个在23种语言下持续提升翻译质量。
- 该研究适用于想理解大模型翻译机制的开发者和研究人员。
稀疏自编码器(SAE)特征被广泛用于解释和引导语言模型行为,但尚不清楚某一语言中发现的特征在另一语言上下文中是否具有相同的因果作用。本文使用翻译启动特征(Wu et al., 2026)研究此问题。我们在Gemma 2中复现并扩展了SAE特征发现方法至多语言场景,涵盖提示语言、源语言与目标语言的变化。通过推理时放大或消融特征激活,测试其对翻译行为的影响。同时考察该方法在Gemma 3中的适用性。结果显示,在两个模型中均发现超过20个跨设置频繁激活的特征,但因果验证表明其中绝大多数影响微弱或不一致。唯有一个特征——Gemma 2的(L10, 5717)和Gemma 3的(L20, 2456)——在23种语言设置下,放大时提升COMET分数,消融时下降,表现一致。结果表明,特征重复出现不能代表跨语言可迁移性,而成功识别出Gemma 2与Gemma 3中一个语言无关的翻译启动方向。
原文摘要 · Abstract (English)
Sparse autoencoder (SAE) features are increasingly used to explain and steer language-model behavior, but it remains unclear whether a feature found in one language context plays the same causal role when processing prompts in another language. We study this question using translation-initiation features (Wu et al., 2026). We reproduce the SAE feature discovery method from Wu et al. in Gemma 2 and extend it to multilingual settings that vary prompt language, source language, and target language. We then test whether features that recur across settings affect translation behavior by amplifying or ablating their activations during inference. We also examine whether the method can be applied to Gemma 3. In both models, we observe an identical finding: although we can find more than 20 features that activate frequently across all discovery settings, causal validation shows that nearly all have small or inconsistent effects. In contrast, one feature -- Gemma 2's (L10, 5717) and Gemma 3's (L20, 2456) -- consistently improves COMET scores when amplified and degrades them when ablated across 23 language settings. These results show that feature recurrence can overstate cross-lingual transfer, while identifying a language-agnostic translation-initiation direction in Gemma 2 and Gemma 3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。