新分词器减少25%分词数量，不降性能
Less Is More: Reducing Token Counts Without Compromising Performance
- 从语料中提取候选词，剔除无效字符和断字错误
- 用概率评分筛选词表，实现分词数减少25%（英）/9%（韩）
- 适合追求推理效率的LLM部署场景
分词方式直接影响大语言模型的推理效率，碎片化分词会增加序列长度与生成成本。虽然更长的多词分词能降低分词频率，但直接使用常导致模型性能下降。本文提出Thunder-Tok，一种子词分词器，在降低分词频率的同时保持下游任务性能。该方法首先从语料子串构建大规模候选词表，并过滤掉结构不完整项，包括无效Unicode字节片段和词边界违规。随后基于训练数据概率的均匀Jensen下界，采用似然得分对候选词表进行剪枝。实验表明，与标准BPE分词器相比，Thunder-Tok在英文上将分词频率降低约25%，在韩文上降低9%，同时保持竞争性性能。
原文摘要 · Abstract (English)
Tokenization directly affects the inference efficiency of large language models, since fragmented tokenization increases sequence length and generation cost. Although longer, multi-word tokens can reduce fertility, naively adding them often degrades language model performance. We propose Thunder-Tok, a subword tokenizer that reduces fertility while preserving downstream performance. Thunder-Tok first constructs a large seed vocabulary from corpus substrings and filters structurally incomplete candidates, including invalid Unicode byte fragments and word-boundary violations. It then prunes the seed vocabulary using a likelihood-based token score derived from a uniform Jensen lower bound of the training-data probability. Experiments show that Thunder-Tok reduces fertility by approximately 25% in English and 9% in Korean compared with the standard BPE tokenizer while maintaining competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。