研究17种印度语言的分词问题，找出更适合低资源复杂语言的分词方法。
Multilingual Tokenization through the Lens of Indian Languages: Challenges and Insights
- 针对11种书写系统、2个语系的印度语言，测试不同分词算法与归一化策略。
- 无损词素边界的效果优于BPE，基于聚类的词汇构建比联合方式更有效。
- 为低资源、形态丰富的语言设计分词器提供实用指导，适合多语言NLP研究者。
分词在自然语言处理中至关重要，是训练语言模型的基础。然而，现有分词器多偏向高资源语言，难以有效处理印度次大陆等语言多样且形态复杂的语言。本文对17种印地语族语言（涵盖11种书写系统和两个语系）进行了系统的多语言分词实证研究。我们系统评估了四种因素的影响：(i) 常用子词算法（BPE与Unigram LM），(ii) 书写系统与正字法感知的归一化，(iii) 词汇量大小，(iv) 多语言词汇构建策略。通过内在与外在评估发现：(i) 书写系统特异性归一化能提升分词质量；(ii) Unigram LM 比 BPE 更好地保留词素边界；(iii) 基于聚类的词汇构建在下游任务中优于联合方法。研究强调了语言学驱动的分词设计的重要性，并为低资源、形态复杂语言的高效分词器构建提供了实践建议。
原文摘要 · Abstract (English)
Tokenization plays a pivotal role in NLP and is fundamental to training language models. However, existing tokenizers are often skewed towards high-resource languages, limiting their effectiveness for linguistically diverse and morphologically rich languages such as those in the Indian subcontinent. In this work, we present a comprehensive empirical study of multilingual tokenization across 17 Indic languages spanning 11 scripts and two language families. We systematically evaluate the effects of (i) widely used subword algorithms: BPE and Unigram LM, (ii) script and orthography-aware normalization, (iii) vocabulary size, and (iv) multilingual vocabulary construction strategies. We use a combination of intrinsic and extrinsic evaluations to obtain the following observations: (i) script-specific normalization improves tokenization quality, (ii) Unigram LM better preserves morphological boundaries than BPE, (iii) cluster-based vocabulary construction shows improvement in downstream tasks compared to the joint method. Our findings highlight the importance of linguistically informed design choices in multilingual tokenization and offer practical guidance for building effective tokenizers for low-resource and morphologically complex languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。