用共享权重让小语种也能高效生成文本
Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages
- 通过共享编码器与解码器权重，复用已有语义空间
- 在四种中国少数民族语言上表现优于更大模型
- 适合资源极低语言的文本生成任务
尽管像XLM-R这样的多语言模型在自然语言处理中推动了多语言发展，但在极低资源语言上的表现仍不理想。现代大模型如LLaMA和Qwen支持的语言数量远少于XLM-R，导致许多语言缺乏文本生成模型。为此，我们提出一种新框架，将多语言编码器适配到极低资源语言的文本生成任务中。通过在编码器和解码器之间共享权重，模型能够利用编码器已学的语义空间，实现高效学习与良好泛化。我们在四种中国少数民族语言上应用该框架，构建了XLM-SWCM，并在多个下游任务中展现出优越性能，甚至超过更大规模模型。
原文摘要 · Abstract (English)
While multilingual language models like XLM-R have advanced multilingualism in NLP, they still perform poorly in extremely low-resource languages. This situation is exacerbated by the fact that modern LLMs such as LLaMA and Qwen support far fewer languages than XLM-R, making text generation models non-existent for many languages in the world. To tackle this challenge, we propose a novel framework for adapting multilingual encoders to text generation in extremely low-resource languages. By reusing the weights between the encoder and the decoder, our framework allows the model to leverage the learned semantic space of the encoder, enabling efficient learning and effective generalization in low-resource languages. Applying this framework to four Chinese minority languages, we present XLM-SWCM, and demonstrate its superior performance on various downstream tasks even when compared with much larger models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。