一个模型,四种速度:150亿参数的超级网络支持实时切换推理速度。
Super Apriel: One Checkpoint, Many Speeds

- 在每层解码器中集成四种注意力混合器,按需切换实现多速率推理。
- 最高提速10.7倍,长文本下吞吐优势更明显,质量损失仅23%。
- 提供自动优化工具,适合需要灵活部署的AI服务开发者。
我们发布 Super Apriel,一个包含150亿参数的超网络,其每个解码器层提供四种训练好的混合器选择——全注意力(FA)、滑动窗口注意力(SWA)、Kimi Delta注意力(KDA)和门控增量网络(GDN)。每层选择一个混合器构成配置,可在服务时即时切换,无需重新加载权重,从而从单一检查点实现多种推理速度。共享检查点还支持无需独立草稿模型的推测性解码。全FA配置在所有报告基准上与Apriel 1.6教师模型性能相当;推荐的混合配置在96%至77%质量保留下,推理吞吐量提升2.9至10.7倍,且长上下文场景下优势进一步放大。48层中每层有四种混合器,配置空间极大。通过基于逐层混合器分配的代理模型预测配置质量,使速度-质量权衡变得可管理,并识别出各速度层级的最佳组合。我们研究了这些最优配置是否能在训练早期确定或仅在收敛后出现。在0.5B规模下排名快速稳定,但在150亿规模下最高效配置表现出更高不稳定性,提示不可从较小模型外推。Super Apriel通过从冻结的Apriel 1.6教师模型进行随机蒸馏,再经监督微调训练完成。我们发布了超网络权重、Fast-LLM训练代码、vLLM服务代码及配置优化工具包。
原文摘要 · Abstract (English)
We release Super Apriel, a 15B-parameter supernet in which every decoder layer provides four trained mixer choices -- Full Attention (FA), Sliding Window Attention (SWA), Kimi Delta Attention (KDA), and Gated DeltaNet (GDN). A placement selects one mixer per layer; placements can be switched between requests at serving time without reloading weights, enabling multiple speed presets from a single checkpoint. The shared checkpoint also enables speculative decoding without a separate draft model. The all-FA preset matches the Apriel 1.6 teacher on all reported benchmarks; recommended hybrid presets span $2.9\times$ to $10.7\times$ decode throughput at 96% to 77% quality retention, with throughput advantages that compound at longer context lengths. With four mixer types across 48 layers, the configuration space is vast. A surrogate that predicts placement quality from the per-layer mixer assignment makes the speed-quality landscape tractable and identifies the best tradeoffs at each speed level. We investigate whether the best configurations at each speed level can be identified early in training or only after convergence. Rankings stabilize quickly at 0.5B scale, but the most efficient configurations exhibit higher instability at 15B, cautioning against extrapolation from smaller models. Super Apriel is trained by stochastic distillation from a frozen Apriel 1.6 teacher, followed by supervised fine-tuning. We release the supernet weights, Fast-LLM training code, vLLM serving code, and a placement optimization toolkit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。