揭示语言模型如何学习语法子结构，发现其损失可线性分解。
Unraveling Syntax: Language Modeling and the Substructure of Grammars
- 通过定义语法子结构，建立语言模型与语法的数学关联
- 模型损失可线性递归分解至不可约子语法，小模型预训练提升性能
- 模型并行学习子结构，优于人类儿童逐步掌握的方式
尽管语言模型表现优异，其学习机制仍不清楚。自然语言语法、编程语言、算术等常由上下文无关文法（CFGs）描述。本文从新视角拓展了神经语言模型对CFG的研究：关注模型对文法子结构（即子语法）的行为。我们定义了子语法，并证明了一系列连接语言建模与子语法的基本定理。结果显示，语言模型的损失在顶层子语法上呈线性递归；进一步递归后，损失可分解为对“不可约”子语法的损失。在额外假设下并通过实证验证，参数化模型能并行学习子语法，不同于儿童先掌握简单结构的模式。子语法预训练可提升小型模型的最终性能，而对齐分析表明，预训练使内部表示更准确反映文法的子结构。
原文摘要 · Abstract (English)
While language models achieve impressive results, their learning dynamics are far from understood. Many domains of interest -- such as natural language syntax, coding languages, arithmetic -- are captured by context-free grammars (CFGs). In this work, we extend prior work on neural language modeling of CFGs in a novel direction: how language modeling behaves with respect to CFG substructure, namely subgrammars. We define subgrammars, and prove a set of fundamental theorems connecting language modeling and subgrammars. We show that language modeling loss recurses linearly over its top-level subgrammars; applied recursively, the loss decomposes into losses for "irreducible" subgrammars. Under additional assumptions, and empirically, parametrized models learn subgrammars in parallel, unlike children who first master simple substructures. We find that subgrammar pretraining can improve final performance, but only for tiny models relative to the grammar, while alignment analyses show that pretraining consistently leads to internal representations that better reflect the grammar's substructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。