大模型对部分整体关系的理解只是半吊子，没真正掌握深层推理。
The quasi-semantic competence of LLMs: a case study on the part-whole relation
- 通过提问、句子概率和向量空间分析三层次测试模型对部分整体关系的认知
- 模型能识别部分整体关系但无法正确判断反事实错误关系，仅具备表面理解
- 适合研究大模型语义能力边界的研究者和对认知语言学感兴趣的学者
理解大型语言模型（LLMs）的语义能力范围与深度是当前人工智能与计算语言学的核心议题。本文聚焦于‘部分-整体’关系（即‘整体论’），该关系在词汇组织中至关重要，却长期被忽视。我们利用ConceptNet关系数据和人类生成的语义特征规范数据，从三个层面分析：(i) 行为测试，直接提示模型回答整体论知识；(ii) 句子概率评分，评估模型对真实与非对称反事实整体论关系的判别能力；(iii) 向量空间分析，验证嵌入与解嵌入空间中整体论概念的线性结构。结果表明，模型对这一关系仅有‘类语义’能力，未能掌握深层推断属性，其知识仍不完整。
原文摘要 · Abstract (English)
Understanding the extent and depth of the semantic competence of \emph{Large Language Models} (LLMs) is at the center of the current scientific agenda in Artificial Intelligence (AI) and Computational Linguistics (CL). We contribute to this endeavor by investigating their knowledge of the \emph{part-whole} relation, a.k.a. \emph{meronymy}, which plays a crucial role in lexical organization, but it is significantly understudied. We used data from ConceptNet relations \citep{speer2016conceptnet} and human-generated semantic feature norms \citep{McRae:2005} to explore the abilities of LLMs to deal with \textit{part-whole} relations. We employed several methods based on three levels of analysis: i.) \textbf{behavioral} testing via prompting, where we directly queried the models on their knowledge of meronymy, ii.) sentence \textbf{probability} scoring, where we tested models' abilities to discriminate correct (real) and incorrect (asymmetric counterfactual) \textit{part-whole} relations, and iii.) \textbf{concept representation} analysis in vector space, where we proved the linear organization of the \textit{part-whole} concept in the embedding and unembedding spaces. These analyses present a complex picture that reveals that the LLMs' knowledge of this relation is only partial. They have just a ``\emph{quasi}-semantic'' competence and still fall short of capturing deep inferential properties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。