MinGram以极简方式实现高效分词，压缩率与语义对齐双优。
MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment
- 基于BPE种子词表，用硬EM和单步剪枝简化训练流程
- 六种语言下压缩率优于BPE与传统Unigram，接近最优词数压缩器
- 适合追求高压缩比且需保留词形结构的NLP任务
Unigram分词器虽便于词汇编辑，但训练复杂耗时。本文提出MinGram（极简Unigram），保留词表表示，通过BPE生成初始词表、在最小词路径上使用硬期望最大化（Hard EM），并仅进行一次全局评分剪枝，省去后缀数组、前向-后向计算及迭代剪枝循环，使训练过程仅需基础分词推理。以词数为首要目标，仅用Unigram得分作为次要选择依据，既保持纯词数方法的压缩能力，又保留概率模型的词形对齐与下游性能。在六种语言上，MinGram压缩效果优于BPE和标准Unigram；压缩优化变体达到最强词数压缩器水平，同时显著提升词形对齐度。在受控的下游语言建模中，包括MinGram在内的Unigram族分词器在比特/字节指标上持续优于BPE。
原文摘要 · Abstract (English)
The Unigram tokenizer uses an elegant representation which makes it straightforward to edit vocabularies, but its training is comparatively heavy and complex. We introduce MinGram (Minimalist Unigram), which keeps the token-list representation but simplifies training using a BPE-derived seed vocabulary, Hard EM on a minimum-token path, and a single flat score-pruning step. This removes the suffix array, the forward-backward pass, and the iterative prune loop, leaving a procedure that requires little beyond tokenizer inference itself. By making token count the primary objective and using a Unigram score only as a tiebreak, MinGram keeps the compression of pure token-count methods while retaining much of the morphological alignment and downstream quality of probabilistic ones. Across six languages, MinGram compresses better than both BPE and standard Unigram, and a compression-oriented variant matches the strongest token-count compressors while retaining substantially higher morphological alignment. In controlled downstream language-model training, Unigram-family tokenizers, with MinGram among the best, consistently beat BPE in bits-per-byte.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。