让模型自己决定如何分词，实现更灵活的语言建模。
From Bytes to Ideas: Language Modeling with Autoregressive U-Nets
- 用自回归U-Net自动学习分词，逐层合并字节为词、短语。
- 深层网络预测未来数个词而非单个字节，捕捉更广语义。
- 可同时处理字符级任务和低资源语言，无需预设分词器。
分词固定了文本的粒度，限制了语言模型对数据的处理方式和预测范围。传统方法如字节对编码（BPE）一次性分割文本，构建静态词汇表，使模型无法改变分词策略。本文提出一种自回归U-Net结构，让模型在训练中自主学习嵌入自己的分词方式：从原始字节开始，逐层聚合为词、词对，直至最多4个词，形成多尺度序列视图。深层阶段需预测更远的未来——不是下一个字节，而是接下来的几个词，因此聚焦于整体语义模式；而浅层则关注细节。通过精细调节预训练计算量，浅层结构达到与强基线BPE相当的表现，深层结构展现出良好潜力。由于分词过程内置于模型内部，该系统既能处理字符级任务，也能跨低资源语言迁移知识。
原文摘要 · Abstract (English)
Tokenization imposes a fixed granularity on the input text, freezing how a language model operates on data and how far in the future it predicts. Byte Pair Encoding (BPE) and similar schemes split text once, build a static vocabulary, and leave the model stuck with that choice. We relax this rigidity by introducing an autoregressive U-Net that learns to embed its own tokens as it trains. The network reads raw bytes, pools them into words, then pairs of words, then up to 4 words, giving it a multi-scale view of the sequence. At deeper stages, the model must predict further into the future -- anticipating the next few words rather than the next byte -- so deeper stages focus on broader semantic patterns while earlier stages handle fine details. When carefully tuning and controlling pretraining compute, shallow hierarchies tie strong BPE baselines, and deeper hierarchies have a promising trend. Because tokenization now lives inside the model, the same system can handle character-level tasks and carry knowledge across low-resource languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。