用字节级接口实现不同分词器间的语言模型蒸馏
Cross-Tokenizer LLM Distillation through a Byte-Level Interface
- 在字节层面统一教师与学生模型的输出分布
- 1B到8B参数模型上表现优于复杂方法
- 为跨分词器知识迁移提供简单有效的新思路
跨分词器蒸馏（CTD）指在教师和学生语言模型使用不同分词器时进行知识迁移，仍是未解决的问题。现有方法依赖启发式策略对齐不匹配的词汇表，引入显著复杂性。本文提出一种简单而有效的基线方法——字节级蒸馏（BLD），通过在所有分词器间共享的字节层面接口实现CTD。具体地，将教师模型的输出分布转换为字节级概率，为学生模型添加轻量级字节级解码头，通过该共享接口进行蒸馏。尽管结构简单，BLD在多个蒸馏任务中表现竞争力，甚至在部分基准上超越更复杂的CTD方法，涵盖1B至8B参数规模的模型。结果表明，字节层是跨分词器知识迁移的自然公共基础；同时，各任务与基准上持续提升仍具挑战，凸显CTD仍是开放问题。
原文摘要 · Abstract (English)
Cross-tokenizer distillation (CTD), the transfer of knowledge from a teacher to a student language model when the two use different tokenizers, remains a largely unsolved problem. Existing approaches rely on heuristic strategies to align mismatched vocabularies, introducing considerable complexity. In this paper, we propose a simple but effective baseline called Byte-Level Distillation (BLD) which enables CTD by operating at a common interface across tokenizers: the byte level. In more detail, we convert the teacher's output distribution to byte-level probabilities, attach a lightweight byte-level decoder head to the student, and distill through this shared byte-level interface. Despite its simplicity, BLD performs competitively with--and on several benchmarks surpasses--significantly more sophisticated CTD methods, across a range of distillation tasks with models from 1B to 8B parameters. Our results suggest that the byte level is a natural common ground for cross-tokenizer knowledge transfer, while also highlighting that consistent improvements across all tasks and benchmarks remain elusive, underscoring that CTD is still an open problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。