用RNN模型在低资源下训练语言模型，效果优于主流Transformer。
BabyHGRN: Exploring RNNs for Sample-Efficient Training of Language Models
- 基于HGRN2架构设计轻量RNN模型BABYHGRN，实现亚二次计算复杂度。
- 在10M和100M词数据集上，多个基准测试表现超越Transformer模型。
- 知识蒸馏提升性能，证明RNN在资源受限场景仍有竞争力。
本文探索了循环神经网络（RNN）及其他亚二次计算复杂度架构在低资源语言建模中的潜力，作为Transformer模型的可行替代方案。我们采用近期提出的HGRN2架构（Qin et al., 2024），对比评估其在BLiMP、EWoK、GLUE和BEAR基准上的表现，结果表明，我们的BABYHGRN语言模型在10M和100M词数据赛道上均优于基于Transformer的基线模型。此外，知识蒸馏进一步提升了性能。研究结果挑战了当前对Transformer架构的过度依赖，表明RNN模型在资源受限环境中依然具备可行性与竞争力。
原文摘要 · Abstract (English)
This paper explores the potential of recurrent neural networks (RNNs) and other subquadratic architectures as competitive alternatives to transformer-based models in low-resource language modeling scenarios. We utilize HGRN2 (Qin et al., 2024), a recently proposed RNN-based architecture, and comparatively evaluate its effectiveness against transformer-based baselines and other subquadratic architectures (LSTM, xLSTM, Mamba). Our experimental results show that BABYHGRN, our HGRN2 language model, outperforms transformer-based models in both the 10M and 100M word tracks of the challenge, as measured by their performance on the BLiMP, EWoK, GLUE and BEAR benchmarks. Further, we show the positive impact of knowledge distillation. Our findings challenge the prevailing focus on transformer architectures and indicate the viability of RNN-based models, particularly in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。