用蒸馏方法把分词模型转成字节级模型，省钱又高效。
Distilling Token-Trained Models into Byte-Level Models
- 分两阶段蒸馏：先对齐字节表示，再端到端字节微调。
- 仅用1250亿字节就保留原模型主要能力。
- 适用于想省训练成本的字节级语言模型研究者。
字节语言模型（BLMs）为突破分词限制提供了新方向，但现有方法需在万亿字节上从头训练，成本高昂。本文提出一种高效蒸馏方案，将已有的分词训练大模型转化为字节级模型，同时保持相近性能。该方案采用两阶段课程：(1) 渐进式知识蒸馏，使字节级表示与分词教师模型的嵌入对齐；(2) 字节级监督微调，实现全程字节空间生成。我们在Llama、Qwen和OLMo等多个模型家族上验证了该方法，结果表明，仅使用约1250亿字节数据，蒸馏后的字节模型即可保留教师模型的大部分能力。
原文摘要 · Abstract (English)
Byte Language Models (BLMs) have emerged as a promising direction for scaling language models beyond tokenization. However, existing BLMs typically require training from scratch on trillions of bytes, making them prohibitively expensive. In this paper, we propose an efficient distillation recipe that converts existing token-trained LLMs into BLMs while retaining comparable capabilities. Our recipe follows a two-stage curriculum: (1) Progressive Knowledge Distillation, which aligns byte-level representations with the embeddings of the token-trained teacher model; and (2) Byte-Level Supervised Fine-Tuning, which enables end-to-end generation entirely in the byte space. We validate our approach across multiple model families, including Llama, Qwen, and OLMo, and demonstrate that the distilled BLMs retain most of the teacher models' performance using only approximately 125B bytes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。