用分层架构实现百万字节序列的高效建模，突破长文本生成瓶颈。
Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling
- 构建分层解码器堆栈，支持单卡全精度训练500万字节上下文。
- 混合架构在训练中高效处理超长序列，生成效率接近线性增长。
- 无需编码器即可在图文问答任务中媲美专用模型，适合多模态基础模型研究。
字节是数字世界的基础，因此是构建多模态基础模型的潜在单元。近期，字节语言模型（BLM）出现以克服分词问题，但字节流过长需新架构支持。为此，我们提出多尺度字节语言模型（MBLM），一种模型无关的分层解码器堆栈，可在单张GPU上以全模型精度训练500万字节的上下文窗口。我们全面评估了基于Transformer和Mamba块的MBLM在单模态与多模态任务上的表现。实验表明，混合架构在训练时能高效处理极长字节序列，并实现近线性的生成效率。据我们所知，这是首个对BLM在视觉问答任务上的评估，结果显示，尽管图像被序列化且无编码器，仅通过纯下一个词预测的MBLM仍可媲美具有特定分类头的定制CNN-LSTM架构。我们还证明，MBLM在整合像素与图像文件流字节等多样化数据表示方面表现出强适应性，凸显其在构建通用多模态基础模型中的潜力。代码已公开于：https://github.com/ai4sd/multiscale-byte-lm
原文摘要 · Abstract (English)
Bytes form the basis of the digital world and thus are a promising building block for multimodal foundation models. Recently, Byte Language Models (BLMs) have emerged to overcome tokenization, yet the excessive length of bytestreams requires new architectural paradigms. Therefore, we present the Multiscale Byte Language Model (MBLM), a model-agnostic hierarchical decoder stack that allows training with context windows of $5$M bytes on single GPU in full model precision. We thoroughly examine MBLM's performance with Transformer and Mamba blocks on both unimodal and multimodal tasks. Our experiments demonstrate that hybrid architectures are efficient in handling extremely long byte sequences during training while achieving near-linear generational efficiency. To the best of our knowledge, we present the first evaluation of BLMs on visual Q\&A tasks and find that, despite serializing images and the absence of an encoder, a MBLM with pure next token prediction can match custom CNN-LSTM architectures with designated classification heads. We show that MBLMs exhibit strong adaptability in integrating diverse data representations, including pixel and image filestream bytes, underlining their potential toward omnimodal foundation models. Source code is publicly available at: https://github.com/ai4sd/multiscale-byte-lm
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。