将基于词元的语言模型转换为字符级模型，提升应用灵活性与压缩效率。
From Language Models over Tokens to Language Models over Characters
- 提出词元级到字符级语言模型的精确与近似转换算法
- 在4个公开模型上实现高精度字符分布近似，且压缩率显著提升
- 适合需要精准控制输入或追求高效压缩的开发者使用
现代语言模型在内部和数学上是词元字符串的分布，而非字符字符串，给基于它们构建用户应用的程序员带来诸多挑战。例如，若提示以字符字符串形式给出，则需先进行词元化才能传递给词元级语言模型，导致分词器及后续处理对提示格式极度敏感（如是否以空格结尾）。本文提出将词元级语言模型转换为字符级模型的算法，包含精确与近似两种方法。在实证部分，我们评估了实际运行时间和近似质量。在四个公开可用的语言模型上，即使计算预算较小，我们的方法仍能以合理速度准确逼近字符级分布，并显著提升语言模型的压缩率（比特/字节）。
原文摘要 · Abstract (English)
Modern language models are internally -- and mathematically -- distributions over $\it{token}$ strings rather than $\it{character}$ strings, posing numerous challenges for programmers building user applications on top of them. For example, if a prompt is specified as a character string, it must be tokenized before passing it to the token-level language model. Thus, the tokenizer and consequent processing are very sensitive to the specification of the prompt (e.g., whether the prompt ends with a space or not). This paper presents algorithms for converting token-level language models to character-level ones. We present both exact and approximate algorithms. In the empirical portion of the paper, we benchmark the practical runtime and approximation quality. Across four publicly available language models, we find that -- even with a small computation budget -- our method is able to accurately approximate the character-level distribution at reasonably fast speeds, and that a significant improvement in the language model's compression rate (bits/byte) is achieved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。