arXiv:2607.17689cs.CL2026-07

给跨语言同形词加语言标记,让模型更好区分不同语言中的相似词。

Tokenizing Crosslingual Homographs

论文配图:Tokenizing Crosslingual Homographs
图 1 · 摘自论文原文
  • 用语言特定符号替换共享词的开头字符,让分词阶段就带语言信息。
  • 实验显示该方法能减少不同语言中同形词被误当作相同词的概率。
  • 对英文转多语翻译有小提升,尤其在BPE分词下效果更明显。

多语言语言模型依赖共享子词词汇表,在有限的词元数量内表示多种语言。这种共享虽有用,但也导致相同表面形式在不同语言中被过于统一处理,即使其含义或用法不同。本文通过跨语言同形词和假朋友现象研究这一局限,并探讨在分词过程中尽早引入语言信息是否能改善处理效果。提出一种简单的分词器级干预:使用语言特定字符替代共享词汇中词的初始字符,从而在词汇构建阶段减少共现同一性。内在分析显示,BPE和UnigramLM通常以语言无关方式处理跨语言同形词,而上下文敏感的SaGe分词器差异更显著;本干预有效缩小了这一差距。在下游英译多语机器翻译任务中,该方法在若干设置下带来小幅提升,尤其在使用BPE时,但效果并非在所有语言和评估集上一致。整体表明,在分词阶段加入轻量级语言信息是值得进一步探索的方向。

原文摘要 · Abstract (English)

Multilingual language models rely on shared subword vocabularies to represent multiple languages within a limited number of token units. While such sharing is often useful, it can also create cases in which identical surface forms are treated too uniformly across languages, even when their meanings or usage differ. We investigate this limitation through cross-lingual homographs and false friends, and examine whether introducing language information earlier in the tokenization process can improve their treatment. We propose a simple tokenizer-level intervention based on language cues: language-specific characters replacing initial characters of shared-vocabulary words, reducing common identity during vocabulary construction. In intrinsic analysis, we find through tokenizer-level statistics that BPE and UnigramLM often treat cross-lingual homographs in a largely language agnostic way, whereas the context-sensitive SaGe tokenizer diverges more strongly; our intervention removes this gap. In downstream English-to-X machine translation, our cues yield modest improvements in several settings, especially under BPE, although the effect is not consistent across all languages and evaluation sets. Overall, the findings suggest that adding lightweight language information at the tokenizer level is a promising direction for further exploration.

多语言分词同形词语言识别

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。