突破空格限制的分词方法，提升模型压缩率与训练效率。
SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
- 通过跨空格分词、渐进式训练和多语言支持，优化分词策略。
- 在10万词汇量下压缩率比标准BPE高17.5%，训练速度提升2.1倍。
- 适用于追求高效分词与多语言建模的研究者与工程师。
分词仍是语言建模中的瓶颈，尤其受限于空格边界。我们提出SupraTok，一种跨空格分词的分词器，包含三个模块：可选的基于熵的数据清洗、基于PMI引导候选搜索的渐进式课程训练，以及多语言脚本处理。在相同未过滤训练数据下，10万词汇量时，SupraTok相比标准BPE压缩率提升17.5%，相比官方SuperBPE提升1.8%，训练速度更快2.1倍。在5万至30万词汇量范围内，始终领先SuperBPE 1.8%–8.6%。单独评估熵过滤：在10万词汇量下使C/T从5.78提升至5.99，而对照组提升较小，SP-BPE-CrossBoundary几乎无变化。在FLORES-200的14种语言上，相比BPE基线取得34.9%的宏平均相对提升。在固定计算量与令牌预算下，使用12层-768维和24层-1024维的GPT-2风格骨干网络（256k词汇量），SupraTok在HellaSwag与MMLU任务中表现更优。结果表明，在公开数据控制比较下，跨空格分词带来一致的压缩增益，熵过滤作为独立流程亦具额外收益。
原文摘要 · Abstract (English)
Tokenization remains a persistent bottleneck in language modeling, especially when vocabulary learning is limited by whitespace boundaries. We present SupraTok, a tokenizer that crosses whitespace boundaries using three modular components: optional entropy-based data curation, staged curriculum training with PMI-guided candidate search, and multilingual script handling. At 100k vocabulary on the same unfiltered training data, SupraTok improves compression over standard BPE by 17.5% and over the official SuperBPE implementation by 1.8%, while training 2.1x faster than SuperBPE. Across 50k-300k vocabularies in the same matched setting, SupraTok remains ahead of SuperBPE by 1.8%-8.6%. We evaluate entropy filtering separately as a pipeline step: at 100k vocabulary it raises SupraTok from 5.78 to 5.99 C/T, while matched controls show a smaller gain for SuperBPE and almost no change for SP-BPE-CrossBoundary. On FLORES-200 across 14 languages, SupraTok yields a macro-averaged 34.9% relative gain over the BPE baseline. In separate downstream experiments with matched compute and fixed token budgets, using 12L-768d and 24L-1024d GPT-2-style backbones with 256k vocabularies, SupraTok improves HellaSwag and MMLU. Overall, these results show that crossing whitespace boundaries gives consistent compression gains under controlled public-data comparisons, while optional entropy filtering provides a separate pipeline benefit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。