提出子词对齐度，更好预测不同文字系统间的跨语言能力。
Beyond Literal Token Overlap: Token Alignability for Multilinguality
- 用子词对齐度替代字面词汇重叠来评估多语言性能。
- 在文字系统差异大的语言对上，对齐度预测效果显著优于传统方法。
- 适合研究跨语言迁移与构建更优多语言分词器的团队使用。
以往研究将词汇重叠或词元分布相似性作为语言模型多语言性和跨语言知识迁移的预测指标。然而，这些字面化的度量方法对使用不同书写系统的语言对赋予高距离值，而这些语言对实际上可能具备良好的跨语言能力，从而削弱了其解释力。本文提出子词对齐度（subword token alignability）作为理解多语言分词影响的新指标。该指标在书写系统差异大、字面词元重叠率低的语言对上，能更准确地预测多语言表现。我们分析了该指标在编码器和解码器模型中的表现，考察了数据量作为潜在干扰因素的影响，并探讨了其在今后多语言分词设计中的应用前景。建议使用此指标识别最优跨语言迁移语言对，并指导未来多语言分词器的构建。代码与可复现性细节已公开。
原文摘要 · Abstract (English)
Previous work has considered token overlap, or even similarity of token distributions, as predictors for multilinguality and cross-lingual knowledge transfer in language models. However, these very literal metrics assign large distances to language pairs with different scripts, which can nevertheless show good cross-linguality. This limits the explanatory strength of token overlap for knowledge transfer between language pairs that use distinct scripts or follow different orthographic conventions. In this paper, we propose subword token alignability as a new way to understand the impact and quality of multilingual tokenisation. In particular, this metric predicts multilinguality much better when scripts are disparate and the overlap of literal tokens is low. We analyse this metric in the context of both encoder and decoder models, look at data size as a potential distractor, and discuss how this insight may be applied to multilingual tokenisation in future work. We recommend our subword token alignability metric for identifying optimal language pairs for cross-lingual transfer, as well as to guide the construction of better multilingual tokenisers in the future. We publish our code and reproducibility details.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。