大模型能识别罕见语法形式，但不懂其含义。
Unpacking Let Alone: Human-Scale Models Generalize to a Rare Construction in Form but not Meaning
- 用自定义数据集测试大模型对罕见语法的掌握能力
- 模型能正确处理形式，但无法理解'let alone'的语义
- 揭示当前模型在形式与意义学习上的不对称性
人类能习得童年中几乎从未见过的语法现象。近期研究表明，具备人规模预训练数据的语言模型可能具备从常见构造泛化到罕见构造的能力。然而，这种泛化能力的范围仍不明确，尤其在罕见构造的意义层面是否有效尚不清楚。本文通过构建专门的合成基准，评估人规模Transformer语言模型对英语罕见且奇特的'let alone'结构在句法和语义两方面的知识。结果发现，即使过滤掉相关构造，大模型仍能感知该结构的形式；但在语义理解上无法做出正确泛化。这表明当前架构在语言形式与意义学习上的样本效率存在不对称，而人类学习者并不存在这一问题。
原文摘要 · Abstract (English)
Humans have a remarkable ability to acquire and understand grammatical phenomena that are seen rarely, if ever, during childhood. Recent evidence suggests that language models with human-scale pretraining data may possess a similar ability by generalizing from frequent to rare constructions. However, it remains an open question how widespread this generalization ability is, and to what extent this knowledge extends to meanings of rare constructions, as opposed to just their forms. We fill this gap by testing human-scale transformer language models on their knowledge of both the form and meaning of the (rare and quirky) English LET-ALONE construction. To evaluate our LMs we construct a bespoke synthetic benchmark that targets syntactic and semantic properties of the construction. We find that human-scale LMs are sensitive to form, even when related constructions are filtered from the dataset. However, human-scale LMs do not make correct generalizations about LET-ALONE's meaning. These results point to an asymmetry in the current architectures' sample efficiency between language form and meaning, something which is not present in human language learners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。