MorphBPE让分词更懂语法结构，提升多语言大模型训练效率。
MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
- 在BPE基础上加入词素信息，实现语义边界感知的分词
- 在4种语言上降低交叉熵损失，加速模型收敛20%以上
- 适合需要高精度分词的多语言NLP任务开发者使用
分词是自然语言处理的基础，直接影响模型效率与语言保真度。尽管字节对编码（BPE）广泛用于大语言模型（LLM），但常忽略词素边界，导致分词效果不佳，尤其在形态丰富的语言中。我们提出MorphBPE，一种融合语言结构的BPE扩展，兼顾统计效率与形态一致性。同时，我们设计两个基于形态的评估指标：(i) 词素一致性F1分数，衡量词素共享与词元共享的一致性，有助于模型训练收敛；(ii) 词素编辑距离，评估词素与词元对齐程度，提升可解释性。在英语、俄语、匈牙利语和阿拉伯语上，针对3亿与10亿参数的LLM进行实验，结果表明MorphBPE持续降低交叉熵损失，加快收敛速度，并显著提升形态对齐得分。MorphBPE完全兼容现有LLM流程，集成仅需少量修改。代码库与分词器试玩平台将开放：https://github.com/llm-lab-org/MorphBPE 和 https://tokenizer.llm-lab.org
原文摘要 · Abstract (English)
Tokenization is fundamental to Natural Language Processing (NLP), directly impacting model efficiency and linguistic fidelity. While Byte Pair Encoding (BPE) is widely used in Large Language Models (LLMs), it often disregards morpheme boundaries, leading to suboptimal segmentation, particularly in morphologically rich languages. We introduce MorphBPE, a morphology-aware extension of BPE that integrates linguistic structure into subword tokenization while preserving statistical efficiency. Additionally, we propose two morphology-based evaluation metrics: (i) Morphological Consistency F1-Score, which quantifies the consistency between morpheme sharing and token sharing, contributing to LLM training convergence, and (ii) Morphological Edit Distance, which measures alignment between morphemes and tokens concerning interpretability. Experiments on English, Russian, Hungarian, and Arabic across 300M and 1B parameter LLMs demonstrate that MorphBPE consistently reduces cross-entropy loss, accelerates convergence, and improves morphological alignment scores. Fully compatible with existing LLM pipelines, MorphBPE requires minimal modifications for integration. The MorphBPE codebase and tokenizer playground will be available at: https://github.com/llm-lab-org/MorphBPE and https://tokenizer.llm-lab.org
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。