用栈增强神经网络实现无需预训练的人类级句法泛化
Bearing Syntactic Fruit with Stack-Augmented Neural Networks
- 用栈结构增强Transformer、RNN等模型，模拟人类句法学习机制
- 在无预训练情况下，带非确定性栈的Transformer在多个任务上表现最佳
- 为语言习得研究提供更贴近人类认知的神经网络模型
儿童学语言时会基于层级规则进行句法归纳。近期研究发现，普通神经网络只有在引入真实句法树、大规模预训练或长期训练后才具备这种归纳能力。本文首次证明，仅通过表面形式训练的栈增强神经网络也能实现类似人类的句法泛化。我们测试了三种基础架构（Transformer、简单RNN、LSTM）搭配两种栈结构，其中一种利用非确定性机制。结果表明，带非确定性栈的Transformer在多个衡量层级归纳偏置的任务中表现最优。这说明栈增强网络可能是比标准架构更准确的人类句法习得模型，对心理语言学研究具有价值。代码已公开。
原文摘要 · Abstract (English)
When children learn language, they make syntactic generalizations based on hierarchical rules. A recent line of work has inquired as to whether common neural network architectures share this inductive bias for hierarchical syntax, finding that they do so only under special conditions: when augmented with ground-truth parse tree structures, when pre-trained on massive corpora, or when trained long past convergence. In this paper, we demonstrate, for the first time, neural network architectures that generalize in human-like fashion when trained only on surface forms: stack-augmented neural networks. We test three base architectures (transformer, simple RNN, LSTM) augmented with two styles of stack, one of which leverages nondeterminism. We find that transformers with nondeterministic stacks generalize best on multiple tasks designed to measure hierarchical inductive bias. This suggests that stack-augmented neural networks may be more accurate models of human syntax acquisition than standard architectures, serving as useful objects of psycholinguistic study. Our code is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。