arXiv:2605.31586cs.CLcs.AI2026-05被引 1

小模型也能理解罕见句式语义,且与常识知识学习相关。

Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions

论文配图:Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions
图 1 · 摘自论文原文
  • 用新数据集测试模型对罕见句式的语义和句法理解能力。
  • 小规模开源模型能准确识别'let alone'等句式的语义,但大模型在语义上表现不佳。
  • 句式语义理解出现晚于句法理解,且与世界知识提升有关,适合研究模型认知机制者参考。

掌握罕见句式(形式-意义配对)的语义一直是个挑战,目前仅最大语言模型能解决。开源模型是否具备稳健的构式理解能力仍不清楚,其学习机制也未明。我们聚焦英语中一组罕见的配对焦点句式(如'let alone', 'much less'),构建新数据集,结合量级形容词语义和通用世界知识测试其意义理解。测试了参数量、架构和预训练数据规模各异的多种模型,发现若干中等规模模型对句式形式和意义均敏感,而人类规模数据训练的模型在所有语义评估中均失败。分析一组开源检查点模型的训练动态发现,配对焦点语义理解出现时间晚于句法理解,且与部分领域世界知识提升相关。整体结果表明,中等规模开源模型可掌握罕见配对焦点句式,且其语义理解与其它意义域知识存在关联。

原文摘要 · Abstract (English)

Grasping the semantics of rare constructions (form-meaning pairings) has been shown to be a challenging problem that has currently only been solved by the largest LLMs. It remains an open question if open-source models have robust constructional understanding, and if so, what learning dynamics underlie the acquisition of this knowledge. Focusing on a set of rare Paired-Focus constructions in English (e.g. "let alone", "much less"), we construct a novel dataset to test their meanings using both scalar adjectival semantics and general world knowledge. Testing a wide range of models differing in parameter count, architecture, and pretraining dataset size, we find that several modestly sized models are sensitive to both the forms and the meanings of Paired-Focus constructions, though models trained on human-scale data fail at all meaning evaluations. Turning to training dynamics for a set of open-checkpoint models, we find that Paired-Focus understanding emerges later in training than Paired-Focus syntactic knowledge, and that learning of Paired-Focus semantics is correlated with gains in some domains of world knowledge. Overall, our empirical results support the conclusion that modestly sized open-source models can grasp the rare Paired-Focus constructions, and demonstrate a connection between knowledge of Paired-Focus constructions and other meaning domains.

句式理解语义学习模型认知常识知识

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。