为孟加拉语设计专用分词器，提升语言模型对复杂词形的处理能力。
Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
- 基于字符级初始化和形态感知合并规则，保持孟加拉语语言一致性。
- 在新闻分类任务中，分词粒度最细且形态可解释性最佳。
- 适合需要高精度处理孟加拉语的NLP研究与应用开发者。
分词是自然语言处理流程中的关键第一步，决定了模型如何学习和表示语言信息。然而，现有的子词分词器（如SentencePiece或HuggingFace BPE）主要针对拉丁语系或多语言语料库设计，在孟加拉语这类形态丰富的语言上表现不佳。为此，我们提出专为孟加拉文设计的BengaliBPE分词器，采用Unicode归一化、字素级初始化和形态感知合并规则，以保持语言一致性并保护子词完整性。通过大规模孟加拉语新闻分类数据集，将BengaliBPE与三种基线方法（空格分词、SentencePiece BPE、HuggingFace BPE）进行比较，评估指标包括分词粒度、编码速度和下游分类准确率。结果显示，尽管BengaliBPE计算开销略高，但在分词细节和形态可解释性方面最优。研究强调了针对形态丰富语言设计专用分词器的重要性，并确立BengaliBPE作为未来孟加拉语NLP系统（包括大模型预训练）的重要基础。
原文摘要 · Abstract (English)
Tokenization is an important first step in Natural Language Processing (NLP) pipelines because it decides how models learn and represent linguistic information. However, current subword tokenizers like SentencePiece or HuggingFace BPE are mostly designed for Latin or multilingual corpora and do not perform well on languages with rich morphology such as Bengali. To address this limitation, we present BengaliBPE, a Byte Pair Encoding (BPE) tokenizer specifically developed for the Bengali script. BengaliBPE applies Unicode normalization, grapheme-level initialization, and morphology-aware merge rules to maintain linguistic consistency and preserve subword integrity. We use a large-scale Bengali news classification dataset to compare BengaliBPE with three baselines: Whitespace, SentencePiece BPE, and HuggingFace BPE. The evaluation considers tokenization granularity, encoding speed, and downstream classification accuracy. While all methods perform reasonably well, BengaliBPE provides the most detailed segmentation and the best morphological interpretability, albeit with slightly higher computational cost. These findings highlight the importance of language-aware tokenization for morphologically rich scripts and establish BengaliBPE as a strong foundation for future Bengali NLP systems, including large-scale pretraining of contextual language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。