小模型新架构，效率与性能双突破
Hymba: A Hybrid-head Architecture for Small Language Models
- 融合注意力与状态空间模型，分头处理记忆与摘要
- 1.5B模型性能超多数同类，缓存缩小11.67倍
- 适合资源受限场景下的高效语言建模
我们提出Hymba，一类小型语言模型家族，采用混合头并行架构，将变换器注意力机制与状态空间模型（SSMs）结合，提升效率。注意力头负责高精度记忆召回，SSM头实现高效上下文压缩。同时引入可学习的元标记，置于提示前以存储关键信息，减轻注意力机制的强制关注负担。通过跨层键值共享和部分滑动窗口注意力进一步优化，显著减小缓存尺寸。在相同条件下对比多种架构的控制实验表明，该设计优势明显。特别地，Hymba-1.5B-Base模型在性能上超越所有小于20亿参数的公开模型，且平均准确率比Llama-3.2-3B高出1.32%，缓存大小减少11.67倍，吞吐量提升3.49倍。
原文摘要 · Abstract (English)
We propose Hymba, a family of small language models featuring a hybrid-head parallel architecture that integrates transformer attention mechanisms with state space models (SSMs) for enhanced efficiency. Attention heads provide high-resolution recall, while SSM heads enable efficient context summarization. Additionally, we introduce learnable meta tokens that are prepended to prompts, storing critical information and alleviating the "forced-to-attend" burden associated with attention mechanisms. This model is further optimized by incorporating cross-layer key-value (KV) sharing and partial sliding window attention, resulting in a compact cache size. During development, we conducted a controlled study comparing various architectures under identical settings and observed significant advantages of our proposed architecture. Notably, Hymba achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model surpasses all sub-2B public models in performance and even outperforms Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size reduction, and 3.49x throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。