用混合结构降低小数据下语言模型的参数量与训练成本
Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures
- 用前馈和类循环网络替代部分注意力层，优化模型结构
- 在Enwik8和Wikitext-103上实现参数量相近时性能更优
- 适合资源受限场景下的高效语言建模应用
基于Transformer的语言模型近年来在文本生成领域取得显著进展，但其高昂的训练成本——参数量达数十亿，计算需求以千万亿次浮点运算/十年计——成为瓶颈。本文研究通过选择性地将注意力层替换为前馈网络和类循环神经网络层，提升低数据环境下模型的性能。我们在标准的Enwik8和Wikitext-103数据集上测试了这些混合架构。结果表明，相较于现有模型，在参数数量相当的情况下，我们的简化架构表现更优；同时，其性能可媲美更大模型，却大幅减少了参数数量。
原文摘要 · Abstract (English)
Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。