比较两种分词方式对多语言模型语法知识的影响。
Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5
- 用相同架构对比子词与字符级分词的效果。
- 模型在中间和后期层编码语法信息，不同语言表现差异大。
- 语法不规则的语言更受益于更多预训练数据。
形态学是多语言建模中的关键因素，直接影响分词策略。本文通过对比mT5与ByT5两个共享架构、目标函数和训练数据的多语言模型，探究分词方式对语法知识编码的影响。二者仅在分词策略上不同：子词分词与字符级分词。在17种语言、4项任务上进行探针分析，结果表明模型对部分语言的形态系统学习效果更好，且形态信息主要存在于模型中后层。此外，语法不规则程度更高的语言，在获得更高比例的预训练数据时表现出更优性能。
原文摘要 · Abstract (English)
Morphology is a crucial factor for multilingual language modeling as it poses direct challenges for tokenization. Here, we seek to understand how tokenization influences the morphological knowledge encoded in multilingual language models. Specifically, we capture the impact of tokenization by contrasting two multilingual language models: mT5 and ByT5. The two models share the same architecture, training objective, and training data and only differ in their tokenization strategies: subword tokenization vs.\@ character-level tokenization. Probing the morphological knowledge encoded in these models on four tasks and 17 languages, our analyses show that the models learn the morphological systems of some languages better than others and that morphological information is encoded in the middle and late layers. Finally, we show that languages with more irregularities benefit more from having a higher share of the pre-training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。