让语言模型逐字生成，解决提示词边界导致的生成偏差问题
Sampling from Your Language Model One Byte at a Time
- 推理时将分词模型转为字节级生成，避免分词带来的语义扭曲
- 解决代码和中文等语言中因分词不匹配引发的生成错误
- 适合需要精准控制生成内容的研究者与开发者
现代语言模型普遍使用分词技术，通过多字节或多字符的标记表示文本，但已有研究指出分词会引入生成畸变，即提示词边界问题（PBP）。例如，用户常被建议不要在提示末尾加空格，否则模型无法将空格纳入下一个标记。虽然该技巧对英语有效，但其根本问题仍影响代码生成及中文等语言，因为这些语言的分词往往不与词法或句法边界对齐。本文提出一种推理时的方法，可将任意采用BPE分词器的自回归语言模型转换为字节级或字符级语言模型。该方法有效缓解了PBP问题，并能统一不同分词器的语言模型词汇表，实现跨模型集成或通过代理微调迁移预训练权重。代码已开源：https://github.com/SewoongLab/byte-sampler。
原文摘要 · Abstract (English)
Tokenization is used almost universally by modern language models, enabling efficient text representation using multi-byte or multi-character tokens. However, prior work has shown that tokenization can introduce distortion into the model's generations, an issue known as the Prompt Boundary Problem (PBP). For example, users are often advised not to end their prompts with a space because it prevents the model from including the space as part of the next token. While this heuristic is effective in English, the underlying PBP continues to affect code generation and languages such as Chinese, where tokens often do not line up with word and syntactic boundaries. In this work, we present an inference-time method to convert any autoregressive LM with a BPE tokenizer into a character-level or byte-level LM. Our method efficiently solves the PBP and is also able to unify the vocabularies of language models with different tokenizers, allowing one to ensemble LMs with different tokenizers at inference time or transfer the post-training from one model to another using proxy-tuning. Code is available at https://github.com/SewoongLab/byte-sampler .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。