6.5亿参数模型实现10亿参数性能，靠的是精巧设计与高效分词。
Supernova: Achieving More with Less in Transformer Architectures
- 采用旋转位置编码与分组查询注意力，提升计算效率。
- 仅用1000亿训练标记，达成10亿参数模型90%性能。
- 适合追求低资源高效率的AI部署场景。
我们提出Supernova，一个6.5亿参数的解码器专用Transformer模型，通过精心的架构设计与分词创新，在保持计算效率的同时达到更大模型的性能。该架构融合旋转位置编码（RoPE）、3:1压缩比的分组查询注意力（GQA）、RMSNorm以提升计算效率，以及SwiGLU激活函数。关键创新在于自研的12.8万词表的字节级BPE分词器，实现了顶尖的压缩性能。详细分析表明，Supernova在仅使用35%参数的情况下，达到10亿参数模型90%的性能，且训练仅需1000亿标记——较同类模型少一个数量级。研究挑战了主流的规模扩张范式，证明架构效率与分词质量可弥补参数量不足。
原文摘要 · Abstract (English)
We present Supernova, a 650M-parameter decoder-only transformer that demonstrates how careful architectural design and tokenization innovation can achieve the performance of larger models while maintaining computational efficiency. Our architecture combines Rotary Positional Embeddings (RoPE), Grouped Query Attention (GQA) with a 3:1 compression ratio, RMSNorm for computational efficiency, and SwiGLU activation functions. A critical innovation is our custom 128,000-vocabulary byte-level BPE tokenizer, which achieves state-of-the-art compression performance. Through detailed analysis, we show that Supernova achieves 90% of the performance of 1B-parameter models while using 35% fewer parameters and requiring only 100B training tokens--an order of magnitude less than competing models. Our findings challenge the prevailing scaling paradigm, demonstrating that architectural efficiency and tokenization quality can compensate for reduced parameter counts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。