用字节级编码直接映射文本，提升处理速度与模型兼容性。
Back to Bytes: Revisiting Tokenization Through UTF-8
- 直接将文本转为UTF-8字节对应的ID，无额外控制符号。
- 提速14倍，数据传输减少8倍，嵌入表仅256×d大小。
- 支持训练后添加位偏置嵌入，推理零成本，适合高效部署。
我们提出UTF8Tokenizer，一种极简的字节级分词器，将文本精确映射为对应其UTF-8编码字节的ID（例如，字节x09对应ID 9）。不同于以往字节级方法（Xue等，2021；Pagnoni等，2025），该实现不引入超出范围的ID（如无ID 256）或辅助符号：所有特殊行为（如填充、边界、对话结构、注意力段、工具调用、“思考”区间等）均通过C0控制字节编码实现，正如原始ASCII设计那样在可打印文本中嵌入控制信息。这些设计带来实际优势：（1）分词速度提升14倍，主机-设备传输量降低8倍（相比int64）；（2）简化且可共享的256×d嵌入表，支持跨模型对齐；（3）通过训练后添加位偏置嵌入，暴露每个字节的位结构，可在训练后加入嵌入表，消除推理开销。该HuggingFace兼容实现提升了语言建模收敛速度。
原文摘要 · Abstract (English)
We present UTF8Tokenizer, a minimalist byte-level tokenizer that maps text exactly to IDs corresponding to the bytes underlying the text's UTF-8 encoding (e.g., byte x09 is token ID 9). Unlike prior byte-level approaches (Xue et al., 2021; Pagnoni et al., 2025), our implementation never introduces out-of-range IDs (i.e. there is no token ID 256) or auxiliary tokens: all special behavior (e.g., padding, boundaries, conversation structure, attention segments, tool calling, "thinking" spans, etc.) is encoded using C0 control bytes - just as ASCII was originally designed to embed control information alongside printable text. These design principles yield practical benefits: (1) faster tokenization (14x) and significantly lower host-device transfer (8x less than int64); (2) simple, shareable 256*d embedding tables that can be aligned across models; and (3) a training-time enhancement via bit-biased embeddings, which exposes per-byte bit structure and can be added to the embedding table post-training, removing inference costs. Our HuggingFace-compatible implementation improves language modeling convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。