动态分词让大模型更高效，跨语言表现更均衡。
Retrofitting Large Language Models with Dynamic Tokenization
- 根据输入内容动态决定分词边界，用融合算法优化子词切分。
- 14种语言平均缩短序列长度超20%，性能下降小于2%。
- 适合追求推理加速和多语言公平性的模型部署场景。
当前语言模型使用固定静态的子词分词器，常导致效率与语言能力下降，尤其在英语以外的语言中更为明显。为此，我们挑战静态设计，提出通过动态分词对大模型进行改造：基于受字节对编码启发的子词合并算法，根据输入文本动态决定分词边界。在批量处理中合并高频子词序列，并利用预训练的嵌入预测超网络实时计算分词嵌入。对于编码器类模型（如XLM-R），该方法在14种语言上平均将序列长度减少超过20%，性能损失低于2%。应用于解码器类模型（如Mistral-7B）的预填充与评分阶段，可实现最高17%的序列长度缩减，且性能几乎无损。总体表明，动态分词能显著缓解静态分词的局限，大幅提升推理速度并促进跨语言公平性，使模型更具适应性与公平性。
原文摘要 · Abstract (English)
Current language models (LMs) use a fixed, static subword tokenizer. This default choice typically results in degraded efficiency and language capabilities, especially in languages other than English. To address this issue, we challenge the static design and propose retrofitting LMs with dynamic tokenization: a way to dynamically decide on token boundaries based on the input text via a subword-merging algorithm inspired by byte-pair encoding. We merge frequent subword sequences in a batch, then apply a pre-trained embedding-prediction hypernetwork to compute the token embeddings on-the-fly. For encoder-style models (e.g., XLM-R), this on average reduces token sequence lengths by >20% across 14 languages while degrading performance by less than 2%. The same method applied to pre-filling and scoring in decoder-style models (e.g., Mistral-7B) results in minimal performance degradation at up to 17% reduction in sequence length. Overall, we find that dynamic tokenization can mitigate the limitations of static tokenization by substantially improving inference speed and promoting fairness across languages, enabling more equitable and adaptable LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。