语言模型生成的文本可能不合法UTF-8，需单独评估其有效性。
Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models
- 用字节级分词训练模型，分离语义与编码结构评估。
- 困惑度稳定在21亿标记后，有效UTF-8生成需42亿标记。
- 罕见字符比常见字符更易生成合法编码，提示高频词过拟合。
字节级分词使语言模型可处理任意Unicode输入，但面对罕见或未见字符时可能生成无效UTF-8序列。我们基于3.55亿参数模型，在包含英语、日语、韩语和中文的均衡多语言语料（共800亿标记）上研究训练规模与UTF-8生成可靠性的关系。引入多种评估协议，将UTF-8结构有效性与语言建模任务解耦。结果表明：困惑度在21亿标记后趋于稳定，而UTF-8有效性需约42亿标记才收敛，滞后约两倍。在无上下文生成中，罕见字符的结构有效性高于常见字符，暗示高频字符表示存在过度特化。实验显示，可靠生成合法UTF-8是独立于困惑度的能力，需专门评估。
原文摘要 · Abstract (English)
Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters. We investigate the relationship between training scale and UTF-8 generation reliability with a 355M parameter model trained on 80B tokens from a balanced multilingual corpus of English, Japanese, Korean, and Chinese. We introduce multiple evaluation protocols that isolate UTF-8 structural validity from language modeling. UTF-8 validity convergence lags perplexity by a roughly a factor of two: perplexity stabilizes after 2.1B tokens, but UTF-8 validity requires 4.2B tokens. In context-free generation, rare characters achieve higher structural validity than common characters, suggesting over-specialization of frequent character representations. Through experiments, we observed that reliable UTF-8 generation is a distinct capability requiring evaluation beyond perplexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。