大模型能组合解析语言，但缺指称基础，理解不如人类深入。
Language Models Can Resolve Reference Compositionally, But It's Not Their Native Strength: The Case of the Personal Relation Task
- 用个人关系任务测试语言模型的组合理解能力
- 模型在结构化语义任务上优于人类，但指称任务差于人类
- 结果表明模型缺乏现实指称锚定，是其理解短板
神经网络模型（如大语言模型）是否真正具备自然语言解释的组合能力？在语义理解中，可区分两个方面：确定表达式在世界中的指称（扩展性任务）和以结构化方式表示其意义（内涵性任务）。我们在个人关系任务（Paperno 2022）中评估了大语言模型与人类在两类任务上的表现。给定一组人及其相互关系，要求解释名词短语如“Amber的父母的朋友”。其中，内涵性任务的答案为公式“friend(parent(amber))”，扩展性任务的答案为具体人物。结果发现：人类在扩展性任务上优于内涵性任务，而大语言模型则相反。该方法揭示了现代机器学习模型组合能力的深层差异。研究支持观点：大语言模型训练中缺乏指称根基，是其难以实现类人语言理解的关键缺失。
原文摘要 · Abstract (English)
Do neural models, such as Large Language Models, genuinely acquire compositional abilities for interpretation of natural language? When we talk about semantic interpretation, we can distinguish two complementary aspects: establishing what an expression refers to in the world (which we call the Extensional task) and representing its sense in a structured way (which we call the Intensional task). We evaluate LLMs and humans on both tasks in the setting of the Personal Relation Task (Paperno 2022) in which, given a universe of people and their relationships with each other, one is asked to interpret a noun phrase such as "Amber's parent's friend". Here, for the Intensional task, the answer is the formula "friend(parent(amber))", and for the Extensional task, the person. We find that humans and LLMs show opposite strengths: humans perform better on Extensional than Intensional tasks, and LLMs vice versa. Our methodology brings greater nuance to the understanding of compositional abilities in modern machine learning models. Our results support the notion that the lack of referential grounding in LLM training is a crucial missing component in mimicking human-like language understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。