70种语言的分词器形态对齐评估发现，对齐程度与模型性能关联不大。
Evaluating Morphological Alignment of Tokenizers in 70 Languages
- 扩展MorphScore至70种语言，评估分词边界与词素边界匹配度。
- 在7种任务上测试5个预训练模型，发现对齐度与性能相关性弱。
- 提示：仅靠形态对齐不能衡量分词器质量，需结合其他指标。
分词是语言建模的关键步骤，影响模型训练与性能，但如何有效评估分词器质量仍不明确。一个关键维度是分词器是否保留语言学上有意义的子词，使分词边界与词素边界对齐。本文将MorphScore（Arnett & Bergen, 2025）从22种语言扩展至70种语言，提升评估灵活性并弥补原版局限。随后，我们针对5个预训练语言模型，在7个下游任务（每种语言至少一个任务）中分析对齐分数与模型性能的相关性。结果表明，形态对齐无法解释模型性能的大部分方差，说明形态对齐本身并非决定模型性能的关键因素。
原文摘要 · Abstract (English)
While tokenization is a key step in language modeling, with effects on model training and performance, it remains unclear how to effectively evaluate tokenizer quality. One proposed dimension of tokenizer quality is the extent to which tokenizers preserve linguistically meaningful subwords, aligning token boundaries with morphological boundaries within a word. We expand MorphScore (Arnett & Bergen, 2025), which previously covered 22 languages, to support a total of 70 languages. The updated MorphScore offers more flexibility in evaluation and addresses some of the limitations of the original version. We then correlate our alignment scores with downstream task performance for five pre-trained languages models on seven tasks, with at least one task in each of the languages in our sample. We find that morphological alignment does not explain very much variance in model performance, suggesting that morphological alignment alone does not measure dimensions of tokenization quality relevant to model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。