用稀疏自编码器发现大模型中语言特异性特征，实现精准语言控制。
Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders
- 通过稀疏自编码器分解模型激活，识别出强关联特定语言的特征。
- 移除特定语言特征仅影响该语言输出，其他语言基本不受影响。
- 多语言协同特征可联合移除，提升语言控制效果，适合可控生成研究者。
大型语言模型（LLMs）的多语言能力机制常通过神经元或内部激活分析，但受叠加效应和层间激活差异影响，可靠性受限。稀疏自编码器（SAEs）能将模型激活分解为稀疏线性组合的特征，我们引入新指标评估特征的单语性，发现部分特征与特定语言强相关。移除这些特征仅显著降低模型在对应语言的表现，对其他语言几乎无影响。有趣的是，某些语言存在多个协同作用的SAE特征，联合移除时性能下降幅度大于单独移除。进一步利用这些语言特异性特征增强引导向量，实现了对模型生成语言的有效控制。代码已开源：https://github.com/Aatrox103/multilingual-llm-features。
原文摘要 · Abstract (English)
The mechanisms behind multilingual capabilities in Large Language Models (LLMs) have been examined using neuron-based or internal-activation-based methods. However, these methods often face challenges such as superposition and layer-wise activation variance, which limit their reliability. Sparse Autoencoders (SAEs) offer a more nuanced analysis by decomposing the activations of LLMs into a sparse linear combination of SAE features. We introduce a novel metric to assess the monolinguality of features obtained from SAEs, discovering that some features are strongly related to specific languages. Additionally, we show that ablating these SAE features only significantly reduces abilities in one language of LLMs, leaving others almost unaffected. Interestingly, we find some languages have multiple synergistic SAE features, and ablating them together yields greater improvement than ablating individually. Moreover, we leverage these SAE-derived language-specific features to enhance steering vectors, achieving control over the language generated by LLMs. The code is publicly available at https://github.com/Aatrox103/multilingual-llm-features.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。