用稀疏自编码器发现大模型中跨语言的语义特异性特征
Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages
- 基于激活概率识别大模型中间层的语言特异性特征
- 这些特征可解释且能实现接近fastText的语言识别性能
- 为理解多语言模型内部机制提供可解释的新路径
理解大语言模型（LLMs）的多语言机制有助于揭示其对不同语言的处理方式，但这一问题仍具挑战性。现有研究多聚焦于单个神经元，但其多义性使得从跨语言表征中分离出语言特异性单元困难。本文探索稀疏自编码器（SAEs）在学习单义特征方面的能力，这些特征能表示大模型中跨语言的具象与抽象概念。尽管部分特征具有语言无关性，语言特异性特征的存在仍待深入研究。为此，我们提出SAE-LAPE方法，基于特征激活概率识别前馈网络中的语言特异性特征。结果表明，许多此类特征主要出现在模型中后层，且具备可解释性。这些特征影响模型的多语言表现和语言输出，可用于语言识别，性能媲美fastText，同时提供更高可解释性。代码与完整图表已开源。
原文摘要 · Abstract (English)
Understanding the multilingual mechanisms of large language models (LLMs) provides insight into how they process different languages, yet this remains challenging. Existing studies often focus on individual neurons, but their polysemantic nature makes it difficult to isolate language-specific units from cross-lingual representations. To address this, we explore sparse autoencoders (SAEs) for their ability to learn monosemantic features that represent concrete and abstract concepts across languages in LLMs. While some of these features are language-independent, the presence of language-specific features remains underexplored. In this work, we introduce SAE-LAPE, a method based on feature activation probability, to identify language-specific features within the feed-forward network. We find that many such features predominantly appear in the middle to final layers of the model and are interpretable. These features influence the model's multilingual performance and language output and can be used for language identification with performance comparable to fastText along with more interpretability. Our code and complete figures are available at https://github.com/LyzanderAndrylie/language-specific-features
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。