让自回归语言模型在不损失准确率的前提下，实现极小词汇量的压缩。
Lossless Vocabulary Reduction for Auto-Regressive Language Models
- 通过理论框架实现无损词汇压缩，保持模型性能不变。
- 可将不同分词方式的模型统一到共同词汇集，提升协作效率。
- 适用于模型集成等场景，尤其适合跨模型协同任务。
分词——将文本分解为子词单位（称为标记）的过程——是语言模型开发中的关键组件。自回归语言模型以逐标记的方式生成文本，即根据前序标记预测下一个标记的分布，因此分词直接影响其生成效率。由于每个语言模型都有自己的词汇表（即可能的标记集合），它们在下一标记分布层面难以协作，例如模型集成。本文建立了一套无损词汇压缩的理论框架，可将任意自回归语言模型转换为具有任意小词汇量的形式，且不损失准确性。该框架使具有不同分词方式的语言模型能高效协作，通过映射到最大公共词汇集实现。我们通过实验证明了其在不同分词方式下的模型集成中的适用性。
原文摘要 · Abstract (English)
Tokenization -- the process of decomposing a given text into a sequence of subwords called tokens -- is one of the key components in the development of language models. Particularly, auto-regressive language models generate texts token by token, i.e., by predicting the next-token distribution given the previous ones, and thus tokenization directly affects their efficiency in text generation. Since each language model has their own vocabulary as a set of possible tokens, they struggle to cooperate with each other at the level of next-token distributions such as model ensemble. In this paper, we establish a theoretical framework of lossless vocabulary reduction, which efficiently converts a given auto-regressive language model into the one with an arbitrarily small vocabulary without any loss in accuracy. This framework allows language models with different tokenization to cooperate with each other efficiently by reduction to their maximal common vocabulary. Specifically, we empirically demonstrate its applicability to model ensemble with different tokenization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。