简单神经网络也能学语法，挑战了大模型必要性
Syntactic Learnability of Echo State Neural Language Models at Scale
- 用大规模回声状态网络做语言建模，结构极简
- 1亿词训练后，语法判断能力媲美甚至超过Transformer
- 适合关注模型效率与最小有效架构的研究者
什么样的神经网络在最低复杂度下仍具备合理语言学习能力？为探索这种简单但充分的语言模型，我们重新考察了一种基础的循环神经网络——回声状态网络（Echo State Network, ESN），它属于一种受限的简单递归神经网络。实验表明，在约1亿词的语料上训练后，具有大隐藏状态的ESN在语法判断任务中的表现可与或优于Transformer，提示在语法学习任务中，Transformer这类复杂架构未必总是必需的。
原文摘要 · Abstract (English)
What is a neural model with minimum architectural complexity that exhibits reasonable language learning capability? To explore such a simple but sufficient neural language model, we revisit a basic reservoir computing (RC) model, Echo State Network (ESN), a restricted class of simple Recurrent Neural Networks. Our experiments showed that ESN with a large hidden state is comparable or superior to Transformer in grammaticality judgment tasks when trained with about 100M words, suggesting that architectures as complex as that of Transformer may not always be necessary for syntactic learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。