提出分层混合架构，用多薄块替代单厚块，提升模型效率与注意力覆盖。
Mixture of Layers with Hybrid Attention
- 用K个细薄块并行替代传统单一大块，通过路由选择激活部分块
- 在8块配置下，模型在1024序列长度上保持93%注意力覆盖率
- 适合追求高效推理的NLP任务，尤其适用于长序列建模
标准Mixture-of-Experts（MoE）Transformer在每层内路由令牌到专家子网络，但层结构仍为单一整体。我们提出分层混合（Mixture of Layers, MoL），将完整的Transformer块（d_model）替换为K个并行的细小块（d_thin << d_model），通过可学习的降维/升维投影连接，并采用top-k块路由进行组合。将稀疏块路由扩展至多个块时，每个块看到的令牌数减少，导致注意力覆盖不足。为此，我们引入混合注意力机制：在共享softmax块中提供全局上下文，同时在路由块中使用门控DeltaNet线性注意力。该设计有效缓解了长序列下的注意力稀疏问题。
原文摘要 · Abstract (English)
Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Layers (MoL), which replaces full-width transformer blocks (d_model) with K parallel thin blocks at reduced dimensionality (d_thin << d_model), connected via learned down/up projections and composed via top-k block routing. Scaling sparse block routing to many blocks creates an attention coverage problem, as each block sees fewer tokens. We address this by introducing hybrid attention, which pairs one shared softmax block for global context with Gated DeltaNet linear attention in routed blocks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。