arXiv:2510.13580cs.CL2025-10

通过精准激活神经元提升低资源语言表现,仅改1%参数却效果显著。

Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models

  • 用信息熵识别语言特异神经元,只微调相关权重。
  • 在12种低资源语言上优于全量微调,仅更新0.2-1%参数。
  • 可新增语言能力且不遗忘旧知识,适合多语言扩展场景。

大型语言模型在不同语言间存在显著性能差异,尤其在高资源与低资源语言之间。本文提出一种框架,通过针对稀疏的语言关联子网络进行定向微调,在保持通用能力的同时提升低资源语言表现。方法利用语言激活概率熵(LAPE)这一信息论指标,可靠捕捉语言特异性激活模式,并仅微调对应权重。在Llama-3.1-8B、Mistral-Nemo-12B和Aya-Expanse-8B上,针对12种中低资源语言的实验表明,该方法持续优于全量微调、仅微调前馈网络、LoRA、IA^3及随机子集基线,且仅更新0.2%-1%参数。进一步证明稀疏神经元微调可在不引发灾难性遗忘的前提下注入新语言能力,具有扩展至其他模型能力的潜力。机制分析显示前馈网络投影在语言适应中起不对称作用,且提升跨语言对齐。最后,我们公开了超过100种语言的语义神经元集合及适配流水线,为低成本扩展大模型至低资源语言提供可行路径。

原文摘要 · Abstract (English)

Large language models (LLMs) exhibit substantial performance disparities across languages, particularly between high- and low-resource settings. We propose a framework for improving performance in underrepresented languages while preserving general-purpose capabilities via targeted fine-tuning of sparse, language-associated subnetworks. Our approach identifies language-relevant neurons using Language Activation Probability Entropy (LAPE), an information-theoretic metric that reliably captures language-specific activation patterns, and fine-tunes only the corresponding weights. Experiments on Llama-3.1-8B, Mistral-Nemo-12B, and Aya-Expanse-8B across 12 mid- and low-resource languages show that our method consistently outperforms full fine-tuning, FFN-only fine-tuning, LoRA, IA^3, and random-subset baselines while updating only 0.2-1% of model parameters. We further show that sparse, neuron-targeted fine-tuning can inject new language capabilities without catastrophic forgetting, with potential applicability to other model capabilities. Mechanistic analyses of weight updates and internal representations reveal asymmetric roles of FFN projections in language adaptation and improved cross-lingual alignment. Finally, we release language neuron sets for over 100 languages together with our adaptation pipeline, enabling a cost-effective path for extending LLMs to underrepresented languages.

语言模型低资源稀疏微调多语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。