无需分词器的端到端语言模型，自动学习字节级抽象表示。
Synergy: End-to-end Concept Model
- 通过学习路由机制，在字节层面实现端到端抽象建模。
- 生成概念标记数少于BBPE，性能相当，且在相同规模下优于Llama3。
- 移除位置编码后中层表现更好，暗示无位置概念的涌现。
本文提出Synergy，一种通过学习路由机制在端到端方式下连接不同抽象层级的语言模型。聚焦低层次语言抽象，我们以字节级语言模型进行训练。模型自发学习对字节进行分词，生成的概念标记数量少于字节级字对编码（BBPE）分词器，同时保持相当的性能。与Llama3对比发现，在相同模型规模和训练数据量下，Synergy具有优势。进一步研究表明，当移除位置编码后，模型中层表现更优，表明出现了位置无关的概念。这些发现验证了无分词器架构的可行性，为更鲁棒、灵活的处理流程开辟了道路。
原文摘要 · Abstract (English)
In this paper, we present Synergy, a language model that bridges different levels of abstraction in an end-to-end fashion through a learned routing mechanism. Focusing on low-level linguistic abstraction, we trained our model as a byte-level language model. Our model spontaneously learns to tokenize bytes, producing fewer concept tokens than Byte-level Byte Pair Encoder (BBPE) tokenizers while keeping comparable performance. By comparing with Llama3, we observed an advantage of Synergy under the same model scale and training dataset size. Further studies show that the middle part (the higher abstraction part) of our model performs better when positional encodings are removed, suggesting the emergence of position-independent concepts. These findings demonstrate the feasibility of tokenizer-free architectures, paving the way for more robust and flexible pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。