改进Transformer的前馈网络结构，能用更少参数更快训练出更好模型。
Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
- 用三层前馈网络替代标准两层结构
- 相同训练时间下损失更低，参数更少
- 适合追求高效训练的NLP研究者
Decoder-only Transformer 模型在语言建模任务中广受欢迎，当前顶尖模型包含上百个变换器块，拥有数十亿可训练参数，并在数万亿词的文本上进行训练。每个变换器块通常由多头注意力（MHA）和双层全连接前馈网络（FFN）组成。本文通过一系列实验考察了前馈网络在模型预训练过程中的重要性，证实其对模型性能具有关键影响。进一步发现，采用三层前馈网络且块数更少的模型配置，能在更短时间内以更少参数实现更低的训练损失，优于标准的两层结构。
原文摘要 · Abstract (English)
Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billions of trainable parameters, and are trained on trillions of tokens of text. Each transformer block typically consists of a multi-head attention (MHA) mechanism and a two-layer fully connected feedforward network (FFN). In this paper, we examine the importance of the FFN during the model pre-training process through a series of experiments, confirming that the FFN is important to model performance. Furthermore, we show that models using a transformer block configuration with three-layer FFNs with fewer such blocks outperform the standard two-layer configuration delivering lower training loss with fewer total parameters in less time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。