对比三种分词法，发现SentencePiece更适配低资源印度语种的零样本命名实体识别。
Tokenization Matters: Improving Zero-Shot NER for Indic Languages
- 用IndicBERT对比BPE、SentencePiece和字级分词在低资源印地语种上的表现。
- SentencePiece在零样本跨语言迁移中准确率更高，且减少未登录词问题。
- 特别适合语法复杂、资源极少的语言如桑塔利语、曼尼普尔语等。
分词是自然语言处理的关键环节，尤其对低资源语言而言，子词分割影响词汇结构与下游任务精度。尽管字节对编码（BPE）是多语言模型的标准分词方法，但其在低资源印地语种（如阿萨姆语、孟加拉语、马拉地语、奥里亚语）及极低资源语种（如桑塔利语、曼尼普尔语、信德语）的命名实体识别（NER）中的适用性尚未充分研究。本文使用IndicBERT系统比较BPE、SentencePiece和字符级分词策略，评估其内在语言特性（分词效率、未登录词率、形态保留）与外在下游性能（微调与零样本跨语言迁移）。结果表明，SentencePiece在低资源印地语种的NER中持续优于BPE，尤其在零样本跨语言设置下表现更优，能更好保持实体一致性。虽然BPE分词最紧凑，但泛化能力差，面对未见语言时常误分类或无法识别实体标签。相比之下，SentencePiece更有效保留语言结构，显著提升桑塔利语、曼尼普尔语等高度形态丰富的语言的实体识别效果，并支持阿拉伯字母书写的信德语等跨书写系统泛化。结果表明，SentencePiece是多语言与低资源印地语种NLP应用中更有效的分词策略。
原文摘要 · Abstract (English)
Tokenization is a critical component of Natural Language Processing (NLP), especially for low resource languages, where subword segmentation influences vocabulary structure and downstream task accuracy. Although Byte Pair Encoding (BPE) is a standard tokenization method in multilingual language models, its suitability for Named Entity Recognition (NER) in low resource Indic languages remains underexplored due to its limitations in handling morphological complexity. In this work, we systematically compare BPE, SentencePiece, and Character Level tokenization strategies using IndicBERT for NER tasks in low resource Indic languages like Assamese, Bengali, Marathi, and Odia, as well as extremely low resource Indic languages like Santali, Manipuri, and Sindhi. We assess both intrinsic linguistic properties tokenization efficiency, out of vocabulary (OOV) rates, and morphological preservation as well as extrinsic downstream performance, including fine tuning and zero shot cross lingual transfer. Our experiments show that SentencePiece is a consistently better performing approach than BPE for NER in low resource Indic Languages, particularly in zero shot cross lingual settings, as it better preserves entity consistency. While BPE provides the most compact tokenization form, it is not capable of generalization because it misclassifies or even fails to recognize entity labels when tested on unseen languages. In contrast, SentencePiece constitutes a better linguistic structural preservation model, benefiting extremely low resource and morphologically rich Indic languages, such as Santali and Manipuri, for superior entity recognition, as well as high generalization across scripts, such as Sindhi, written in Arabic. The results point to SentencePiece as the more effective tokenization strategy for NER within multilingual and low resource Indic NLP applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。