提出字节级模型可分离语言建模与分块边界，实现无损知识迁移。
Disentangling Language Modeling and Boundaries
- 字节级模型共享输出空间，可精确转移知识
- 实验验证语言建模与边界分布可独立调整
- 适合希望免去分词器束缚的研究者
字节级语言模型通常被认为具备鲁棒性、多语言公平性和字符级能力。本文指出其另一结构优势：因以字节为读写单位，任意两个字节级模型共享输出空间，知识迁移可完全精确且不受原始分词方式影响。我们假设，字节级模型生成的两个分布——下一字节的概率分布、分块边界的位置分布——可被解耦并几乎独立调整。模型可吸收教师的能力而保留自身边界，或改变边界设置而保持原有能力。本文提出两项实验以验证该假设，并给出相关性质的初步测量结果。我们认为，社区应转向字节级接口作为通用标准：若假设成立，则一旦字节级模型成为主流，模型间的能力迁移与边界重构将变得廉价且常规，摆脱当前各模型专属分词器的限制。
原文摘要 · Abstract (English)
Byte-level language models are usually argued for on the grounds of robustness, multilingual fairness, and character-level skills. We point to a different, structural advantage: because they read and write bytes, any two of them share an output space, so knowledge transfer between them is exact and independent of how either was originally tokenized. We hypothesize that the two distributions a byte-level model produces, one over the next byte, one over where its patch boundaries fall, can be disentangled and changed almost independently. A model could absorb a teacher's capability while keeping its own boundaries, or change how it places those boundaries while keeping its capabilities. We lay out the two experiments that would settle the hypothesis, alongside preliminary measurements of the properties they rest on. We argue that the community should move toward a byte-level interface as a shared standard: if the hypothesis holds, then once byte-level models are the norm, transferring capabilities and reshaping boundaries between them become cheap and routine, free of the per-model tokenizer that blocks them today.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。