多语言模型训练中，神经元从分语言表征逐步演化为跨语言共享表示。
Cross-Lingual Generalization and Compression: From Language-Specific to Shared Neurons
- 分析三个多语言模型参数空间，发现语言特异性表征随训练逐渐压缩为跨语言抽象。
- 深层神经元在预训练中对语义概念的编码逐渐对齐，实现跨语言一致性。
- 识别出在多语言中稳定预测相同概念的关键神经元，适合研究模型泛化机制者阅读。
多语言语言模型（MLLMs）虽未接受显式跨语言监督，却展现出强大的跨语言知识迁移能力。我们分析了三个MLLM的参数空间，研究其在预训练过程中表征的演变规律，观察到与压缩一致的模式：模型初期形成语言特异性表征，随着训练推进，这些表征逐渐收敛为跨语言抽象。通过探测实验，我们发现模型各层的语言识别能力从均匀分布演变为更专一的功能分工。为进一步分析，我们聚焦于编码特定语义概念的神经元，追踪其在预训练中的发展，结果表明它们在不同语言间逐步对齐。值得注意的是，我们识别出一些神经元在跨语言中日益成为相同概念的可靠预测器。
原文摘要 · Abstract (English)
Multilingual language models (MLLMs) have demonstrated remarkable abilities to transfer knowledge across languages, despite being trained without explicit cross-lingual supervision. We analyze the parameter spaces of three MLLMs to study how their representations evolve during pre-training, observing patterns consistent with compression: models initially form language-specific representations, which gradually converge into cross-lingual abstractions as training progresses. Through probing experiments, we observe a clear transition from uniform language identification capabilities across layers to more specialized layer functions. For deeper analysis, we focus on neurons that encode distinct semantic concepts. By tracing their development during pre-training, we show how they gradually align across languages. Notably, we identify specific neurons that emerge as increasingly reliable predictors for the same concepts across languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。