arXiv:2605.08301cs.LGcs.AI2026-05被引 4

用预训练模型快速构建高效混合模型,实现长文本推理性能提升。

Priming: Hybrid State Space Models From Pre-trained Transformers

论文配图:Priming: Hybrid State Space Models From Pre-trained Transformers
图 1 · 摘自论文原文
  • 从预训练Transformer迁移知识,仅用不到0.5%的训练量完成混合模型初始化
  • 32B模型在长文本推理上比原模型高3.8分,解码速度提升2.3倍
  • 支持多种模型架构与规模,可对比不同状态空间模块性能优劣

混合状态空间模型将注意力机制与循环状态空间模型(SSM)层结合,兼顾注意力带来的瞬时记忆与SSM的压缩衰减记忆,从而减少键值缓存、加速解码,并拓展了更丰富的架构设计空间。然而,此前探索该空间需从头训练,限制了大规模对比研究。本文提出Priming方法,将混合模型设计转化为知识迁移问题:从预训练Transformer初始化,经短周期对齐与微调,仅使用不足0.5%的源模型预训练数据量即可恢复下游性能。Priming不依赖源模型家族(如Qwen、Llama、Mistral)、模型类型(密集或专家混合)或规模。我们首次在相同条件下系统比较了Gated KalmanNet(GKA)、Gated DeltaNet(GDN)和Mamba-2三种SSM模块,发现其表达能力排序为GKA>GDN>Mamba-2,且直接预测长上下文推理表现。将Priming扩展至8B/32B推理模型,支持128K原生上下文。所提混合GKA 32B模型在平均推理得分上较源Qwen3-32B提升+3.8分,性能接近同数据微调的Transformer(差距<1%),同时实现最高2.3倍解码吞吐。为推动研究,我们开源了针对长上下文推理与指令遵循的多款预训练混合模型,以及包含序列并行训练、优化GKA内核与vLLM服务插件在内的完整代码,均采用Apache 2.0许可。

原文摘要 · Abstract (English)

Hybrid State-Space models combine Attention with recurrent State-Space Model (SSM) layers, balancing eidetic memory from Attention with compressed fading memory from SSMs. This yields smaller Key-Value caches and faster decoding than Transformers, along with a richer architectural design space. Exploring that design space at scale has so far required training from scratch, a barrier that has kept most large-model Hybrid research within a narrow range of architectures. We introduce Priming, a method that turns Hybrid architecture design from a pre-training problem into a knowledge transfer one. Priming initializes a Hybrid model from a pre-trained Transformer and, through short alignment and post-training phases, recovers downstream quality using less than 0.5% of the source model's pre-training token budget. Priming is agnostic to the source Transformer family (e.g., Qwen, Llama, Mistral), model class (dense or Mixture-of-Experts), and model scale. Priming enables us to run the first controlled comparison of SSM layer types at scale under identical conditions. We evaluate, Gated KalmaNet (GKA), Gated DeltaNet (GDN), and Mamba-2, and show that their expressiveness hierarchy, GKA>GDN>Mamba-2, directly predicts downstream performance on long-context reasoning tasks. We scale Priming to 8B/32B reasoning models with native 128K contexts. Our Hybrid GKA 32B improves over its source Qwen3-32B by +3.8 average reasoning points, while staying within 1% of a Transformer post-trained on the same data and enabling up to 2.3x higher decode throughput. To foster research on Hybrid architectures, we release a model zoo of primed Hybrid models for long-context reasoning and instruction following, together with the Priming training and inference code (Sequence Parallelism algorithms for long-context training, optimized GKA kernels, and vLLM serving plugin), all under Apache~2.0 License.

混合模型状态空间长文本模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。