arXiv:2603.03583cs.CLcs.LG2026-03被引 1

无需分词器,模型自适应压缩字节流生成语义单元

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer

  • 通过压缩率动态分割原始字节流,实现无固定分词的自适应建模
  • 在多个任务上超越BPE模型和传统字节级架构,性能显著提升
  • 适合追求端到端、信息驱动语言建模的研究者与应用

当前语言模型仍依赖固定的子词分词器。一旦训练完成,模型只能以固定粒度运行,常导致脆弱且反直觉的行为,即使在强大的推理模型中也是如此。我们提出 extbf{ByteFlow Net},一种新层次化架构,彻底摒弃分词器,使模型能自主将原始字节流分割为有意义的语义单元。该方法基于潜在表示的编码率进行压缩驱动的分段,生成自适应边界,同时通过Top-K选择保持静态计算图。不同于依赖人类设计启发式规则的先前自分词方法,ByteFlow Net 能根据输入内容动态调整内部表征粒度。实验表明,这种基于压缩的分块策略带来显著性能提升:ByteFlow Net 在多项任务上优于基于BPE的Transformer及以往字节级架构。结果表明,端到端、无分词器建模不仅可行,而且更有效,为构建更具适应性和信息基础的语言模型开辟了新路径。

原文摘要 · Abstract (English)

Modern language models still rely on fixed, pre-defined subword tokenizations. Once a tokenizer is trained, the LM can only operate at this fixed level of granularity, which often leads to brittle and counterintuitive behaviors even in otherwise strong reasoning models. We introduce \textbf{ByteFlow Net}, a new hierarchical architecture that removes tokenizers entirely and instead enables models to learn their own segmentation of raw byte streams into semantically meaningful units. ByteFlow Net performs compression-driven segmentation based on the coding rate of latent representations, yielding adaptive boundaries \emph{while preserving a static computation graph via Top-$K$ selection}. Unlike prior self-tokenizing methods that depend on brittle heuristics with human-designed inductive biases, ByteFlow Net adapts its internal representation granularity to the input itself. Experiments demonstrate that this compression-based chunking strategy yields substantial performance gains, with ByteFlow Net outperforming both BPE-based Transformers and previous byte-level architectures. These results suggest that end-to-end, tokenizer-free modeling is not only feasible but also more effective, opening a path toward more adaptive and information-grounded language models.

语言模型自分词无分词器压缩建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。