研究发现，多语言模型中词汇重叠有助于跨语言迁移。
False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models
- 通过控制词汇重叠程度，系统对比不同设置下的模型表现。
- 重叠词汇越多，跨语言任务性能越优，尤其在XNLI和XQuAD上。
- 共享词汇的语义相似性增强跨语言表征能力，适合多语言模型设计。
在多语言语料上训练的子词分词器会自然产生跨语言的重叠分词。这种重叠是促进跨语言迁移，还是引入语言间干扰？以往研究结果不一，部分源于实验设置差异及分词频率、粒度等混杂因素。为此，我们设计了一项受控实验，系统地在多个语言对上训练双语自回归模型，考察不同词汇重叠水平的影响。关键在于，我们探索了重叠词汇的语义相似性如何影响迁移效果。分析隐藏表示发现，任何形式的重叠都能使嵌入空间捕捉跨语言语义关系，而无重叠模型则弱得多。在XNLI和XQuAD上的实验表明，有重叠的模型优于无重叠模型，且随着重叠程度增加，迁移性能普遍提升。总体而言，研究揭示了词汇重叠的优势，表明显著的共享词汇仍是多语言分词器的重要设计选择。
原文摘要 · Abstract (English)
Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages. Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages? Prior work offers mixed evidence, partly due to varied setups and confounders, such as token frequency or subword segmentation granularity. To address this question, we devise a controlled experiment where we train bilingual autoregressive models on multiple language pairs under systematically varied vocabulary overlap settings. Crucially, we explore a new dimension to understanding how overlap affects transfer: the semantic similarity of tokens shared across languages. We first analyze our models' hidden representations and find that overlap of any kind creates embedding spaces that capture cross-lingual semantic relationships, while this effect is much weaker in models with disjoint vocabularies. On XNLI and XQuAD, we find that models with overlap outperform models with disjoint vocabularies, and that transfer performance generally improves as overlap increases. Overall, our findings highlight the advantages of token overlap in multilingual models and show that substantial shared vocabulary remains a beneficial design choice for multilingual tokenizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。