发现中间层的前馈网络对预训练至关重要，集中配置更优。
Layerwise Importance Analysis of Feed-Forward Networks in Transformer-based Language Models
- 通过增删不同层的前馈网络，测试其在预训练中的重要性。
- 70%连续中层集中前馈网络，下游任务表现最优。
- 适用于想优化模型结构、提升效率的研究者。
本研究探究了基于Transformer的语言模型在预训练过程中各层前馈网络（FFNs）的重要性。提出一种实验方法：在保持总参数量不变的前提下，增加部分层的FFN维度，完全移除其他层的FFN。由于关注的是预训练阶段的FFN重要性，我们从零开始训练模型，考察其重要性是否随层位置变化，而非使用公开的预训练模型。通过对多种规模（285M、570M、1.2B参数）和层数（12、24、40层）的模型进行综合评估，结果表明，将前馈网络集中在70%的连续中间层，能持续优于标准配置，在多个下游任务上表现更佳。
原文摘要 · Abstract (English)
This study investigates the layerwise importance of feed-forward networks (FFNs) in Transformer-based language models during pretraining. We introduce an experimental approach that, while maintaining the total parameter count, increases the FFN dimensions in some layers and completely removes the FFNs from other layers. Furthermore, since our focus is on the importance of FFNs during pretraining, we train models from scratch to examine whether the importance of FFNs varies depending on their layer positions, rather than using publicly available pretrained models as is frequently done. Through comprehensive evaluations of models with varying sizes (285M, 570M, and 1.2B parameters) and layer counts (12, 24, and 40 layers), we demonstrate that concentrating FFNs in 70% of the consecutive middle layers consistently outperforms standard configurations for multiple downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。