通过分析神经元级语言特性，实现多语言大模型高效扩展。
NeuronMoE: Neuron-Guided Mixture-of-Experts for Efficient Multilingual LLM Extension
- 基于神经元层面的跨语言多样性，动态分配每层专家数量。
- 在希腊语、土耳其语等低资源语言上节省40%参数，性能相当。
- 发现低资源语言专家也呈现早期与晚期层的神经特化模式。
将大语言模型扩展至低资源语言对全球可及性至关重要，但为每种语言训练独立模型成本过高。混合专家（MoE）架构通过添加稀疏的语言特定参数解决此问题，但每层应配置多少专家仍是未解难题。现有方法依据层级相似性分配专家，然而语言处理在单个神经元层面存在精细专化。我们提出NeuronMoE，通过分析所有Transformer组件中的语言特异神经元，根据实测的跨语言神经元多样性指导每层专家分配。应用于Llama-3.2-3B支持希腊语、土耳其语和匈牙利语时，平均参数减少约40%，性能与LayerMoE基线相当。我们发现低资源语言专家会独立发展出类似高资源语言的神经特化模式，且集中于早期和晚期层，揭示了多语言模型组织语言知识的潜在通用架构规律。
原文摘要 · Abstract (English)
Extending large language models to low-resource languages is essential for global accessibility, but training separate models per language is prohibitively expensive. Mixture-of-Experts (MoE) architectures address this by adding sparse language-specific parameters, but determining how many experts each layer needs remains an open question. Current approaches allocate experts based on layer-level similarity, yet language processing exhibits fine-grained specialization at individual neurons. We propose $\textbf{NeuronMoE}$, a method that analyzes language-specific neurons across all transformer components to guide expert allocation per layer based on empirically measured cross-lingual neuron diversity. Applied to Llama-3.2-3B for low-resource languages (Greek, Turkish, and Hungarian), this approach achieves approximately 40% average parameter reduction while matching the performance of the LayerMoE baseline. We find that low-resource language experts independently develop neuron specialization patterns mirroring the high-resource language, which are concentrated in early and late layers. This reveals potential universal architectural principles in how multilingual models organize linguistic knowledge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。