用动态字节块替代固定分词，提升大模型效率与泛化能力。
Byte Latent Transformer: Patches Scale Better Than Tokens
- 字节按熵动态分块，复杂处分配更多算力
- 80亿参数模型在4万亿字节上训练，推理效率提升
- 适合追求高效推理和长尾泛化的研究者
我们提出字节潜在变换器（BLT），一种全新的字节级大语言模型架构。首次在规模上达到基于分词的LLM性能，同时显著提升推理效率与鲁棒性。BLT将字节编码为动态大小的块，作为计算主单元，块的分割依据下一字节熵值，数据复杂度高时自动分配更多算力与模型容量。我们首次完成从80亿参数到4万亿训练字节的字节级模型的受控浮点运算量缩放研究。结果表明，无需固定词汇表即可实现原始字节训练的可扩展性。由于在可预测数据中动态选择长块，训练与推理效率均得到提升，并在推理能力和长尾泛化上展现定性改进。在固定推理成本下，通过同时增大块与模型规模，BLT展现出远超分词模型的扩展优势。
原文摘要 · Abstract (English)
We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。