用信息熵指导中文分词，提升BPE分词准确率
Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
- 用信息熵和互信息识别词语边界，引导BPE分词
- 在PKU数据集上F1分数显著优于标准BPE
- 适合低资源与多语言场景的分词优化
字节对编码（BPE）因其简单性和在下游任务中的优异表现，已成为现代语言模型中广泛采用的子词分词方法。然而，将BPE应用于无空格语言（如中文）时面临挑战，因其频率驱动的合并机制忽略了语言学边界。为此，我们提出两种基于信息熵的预分词策略，利用无监督信息论线索引导BPE分词。第一种方法结合点互信息与左右熵识别连贯字符片段；第二种方法则使用预训练GPT-2模型的预测熵检测边界不确定性。我们在PKU数据集的一个子集上评估了两种方法，结果表明其在分词精度、召回率和F1分数上均显著优于标准BPE。结果表明，熵引导的预分词不仅提升了与标注语言单位的对齐度，也为低资源与多语言场景下的分词质量改进提供了可行方向。
原文摘要 · Abstract (English)
Byte-Pair Encoding (BPE) has become a widely adopted subword tokenization method in modern language models due to its simplicity and strong empirical performance across downstream tasks. However, applying BPE to unsegmented languages such as Chinese presents significant challenges, as its frequency-driven merge operation is agnostic to linguistic boundaries. To address this, we propose two entropy-informed pre-tokenization strategies that guide BPE segmentation using unsupervised information-theoretic cues. The first approach uses pointwise mutual information and left/right entropy to identify coherent character spans, while the second leverages predictive entropy derived from a pretrained GPT-2 model to detect boundary uncertainty. We evaluate both methods on a subset of the PKU dataset and demonstrate substantial improvements in segmentation precision, recall, and F1 score compared to standard BPE. Our results suggest that entropy-guided pre-tokenization not only enhances alignment with gold-standard linguistic units but also offers a promising direction for improving tokenization quality in low-resource and multilingual settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。