Aya-23多语言模型内部如何处理混合语言，揭示其独特表征机制。
What Language(s) Does Aya-23 Think In? How Multilinguality Affects Internal Language Representations
- 通过激活模式分析，发现其翻译时依赖语系关联而非单一英语枢纽。
- 混合语言输入中，底层语言主导神经元激活，且最终层集中特异性神经元。
- 模型内部表征受文字脚本和语言类型关系影响，适合跨语言研究者参考。
大型语言模型（LLMs）在多语言任务上表现优异，但其内部语言处理机制仍不清晰。本文分析了基于均衡多语言数据训练的 decoder-only 模型 Aya-23-8B，对比 Llama 3 与 Chinese-LLaMA-2 等以单语为主的模型，在代码混合、填空及翻译任务中的表现。通过 logit lens 和神经元专一性分析发现：(1) Aya-23 在翻译时激活语系相关语言表征，不同于英语中心模型依赖单一枢纽语言；(2) 代码混合输入的神经元激活模式随混合比例变化，且更受基础语言影响；(3) 针对混合输入的语言特异性神经元集中在最后几层，与以往 decoder-only 模型结论不同。神经元重叠分析进一步显示，书写系统相似性与语言类型关系影响跨模型处理。这些结果揭示多语言训练如何塑造模型内部表示，为未来跨语言迁移研究提供依据。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at multilingual tasks, yet their internal language processing remains poorly understood. We analyze how Aya-23-8B, a decoder-only LLM trained on balanced multilingual data, handles code-mixed, cloze, and translation tasks compared to predominantly monolingual models like Llama 3 and Chinese-LLaMA-2. Using logit lens and neuron specialization analyses, we find: (1) Aya-23 activates typologically related language representations during translation, unlike English-centric models that rely on a single pivot language; (2) code-mixed neuron activation patterns vary with mixing rates and are shaped more by the base language than the mixed-in one; and (3) Aya-23's languagespecific neurons for code-mixed inputs concentrate in final layers, diverging from prior findings on decoder-only models. Neuron overlap analysis further shows that script similarity and typological relations impact processing across model types. These findings reveal how multilingual training shapes LLM internals and inform future cross-lingual transfer research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。