提出混合卷积架构，在大规模模型上实现比Transformer更快的训练速度。
Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
- 结合卷积与注意力，按任务需求动态选择算子，提升效率。
- 400亿参数下训练速度比优化版Transformer快1.2至2.9倍。
- 适合需要高速训练的大规模序列建模任务，尤其字节级分词场景。
我们提出卷积式多混合架构，基于两个简单观察：其一，混合模型中的算子可针对上下文召回、多标记召回和压缩等任务定制，输入相关的卷积与注意力提供互补优势；其二，协同设计卷积算子与硬件感知算法，在以往架构难以超越Transformer的场景中实现效率提升。在400亿参数规模下，端到端训练速度比优化版Transformer快1.2至2.9倍，也比前代混合模型快1.1至1.4倍。在H100 GPU和模型宽度4096条件下，所提多混合StripedHyena 2架构中各算子吞吐量相比线性注意力和状态空间模型提升两倍。多混合架构在字节级分词数据上的序列建模表现优异，如Evo 2系列模型所示。我们讨论了实现这些结果的基础，包括架构设计、用于张量核心的重叠-加阻塞内核，以及专用的全对全和点对点上下文并行策略。
原文摘要 · Abstract (English)
We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression, with input-dependent convolutions and attention offering complementary performance. Second, co-designing convolution operators and hardware-aware algorithms enables efficiency gains in regimes where previous alternative architectures struggle to surpass Transformers. At the 40 billion parameter scale, we train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids. On H100 GPUs and model width 4096, individual operators in the proposed multi-hybrid StripedHyena 2 architecture achieve two-fold throughput improvement over linear attention and state-space models. Multi-hybrids excel at sequence modeling over byte-tokenized data, as demonstrated by the Evo 2 line of models. We discuss the foundations that enable these results, including architecture design, overlap-add blocked kernels for tensor cores, and dedicated all-to-all and point-to-point context parallelism strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。