用MLP块替换顶层注意力层，显著提升生成速度与吞吐量。
ShishuLM : Achieving Optimal and Efficient Parameterization with Low Attention Transformer Models
- 顶层用MLP块替代完整解码层，减少计算开销。
- 生成延迟降低10%-60%，吞吐量提升1.3-5倍。
- 相邻MLP层共享参数，内存节省20%且性能几乎不变。
尽管Transformer架构在自然语言处理任务中表现优异，但其带来巨大内存和计算开销。近期研究发现模型顶层注意力子层存在显著结构冗余，为优化提供了可能。受推理时层剪枝和深度依赖计算启发，本文提出高效语言模型ShishuLM：将模型顶层的完整解码层替换为仅含MLP的模块，实现生成延迟降低10%-60%，吞吐量提升1.3-5倍。进一步通过共享相邻MLP块的参数，内存使用最多减少20%，性能损失极小。研究结果为从预训练角度构建更高效的语言建模架构提供了新思路，揭示了Transformer中信息流动的本质。
原文摘要 · Abstract (English)
While the transformer architecture has achieved state-of-the-art performance on natural language processing tasks, these models impose substantial memory and computational overhead. Recent research has identified significant architectural redundancies within these models, particularly in the attention sub-layers in the top layers, presenting opportunities for optimization without compromising performance. Taking insights from research on inference-time layer pruning and depth-dependent computation in language models, we introduce an efficient language model architecture referred to as ShishuLM. By replacing full decoder layers at the top of the model with MLP-only blocks, we achieve up to 10-60% improvement in generation latency and 1.3 -5 $\times$ gain in throughput. Upon further sharing parameters across adjacent MLP-only layers of ShishuLM, we obtain up to 20% savings in memory with minimal degradation in performance. Our findings provide insights towards building more efficient language modeling architectures from a pre-training standpoint by leveraging how information flows in transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。