用双语词典实现低资源语言词汇迁移，提升跨语言模型性能。
Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries
- 基于BPE分词器特性，通过逐步移除子词重构目标语言嵌入。
- 在多个低资源语言上优于现有方法，最高提升12.3个点。
- 适合无大规模语料但有词典的冷门语言场景。
跨语言词汇迁移在将预训练语言模型适配到新语言（包括低资源语言）方面具有潜力。现有依赖单语或平行语料的方法在资源有限的语言上面临挑战。本文提出一种简单而有效的方法，利用广泛存在的双语词典（得益于描述语言学家的工作）。该方法利用BPE分词器的一个特性：从词汇表中移除某个子词会导致回退到更短的子词。通过迭代地逐步移除目标子词，估算其嵌入表示。实验结果表明，该方法在低资源语言上的表现优于现有方法，证明了基于词典的跨语言词汇迁移的有效性。
原文摘要 · Abstract (English)
Cross-lingual vocabulary transfer plays a promising role in adapting pre-trained language models to new languages, including low-resource languages. Existing approaches that utilize monolingual or parallel corpora face challenges when applied to languages with limited resources. In this work, we propose a simple yet effective vocabulary transfer method that utilizes bilingual dictionaries, which are available for many languages, thanks to descriptive linguists. Our proposed method leverages a property of BPE tokenizers where removing a subword from the vocabulary causes a fallback to shorter subwords. The embeddings of target subwords are estimated iteratively by progressively removing them from the tokenizer. The experimental results show that our approach outperforms existing methods for low-resource languages, demonstrating the effectiveness of a dictionary-based approach for cross-lingual vocabulary transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。