对比人类与大模型对隐含信息的理解差异
Comparing Human and Large Language Model Interpretation of Implicit Information
- 构建知识图谱提取上下文中的关系三元组，验证隐含推理
- 模型覆盖范围有限，人类常提出更多隐含推论
- 模型在社交语境中更保守，人类在简短语境中更谨慎
隐含意义的解读是人类交流的核心部分，但这一能力未必适用于与大语言模型（LLMs）的交互。为此，我们提出了隐含信息抽取（IIE）任务，并设计了一个基于LLM的IIE流程：从上下文句子构建结构化知识图谱，提取关系三元组，验证隐含推断，并分析时间关系。我们在两个数据集上评估了两种LLM，对比其与众包人类判断的结果。发现人类与模型在多数三元组上达成一致，但人类持续提出大量新增推论，表明当前基于LLM的IIE覆盖不足。此外，在社会性丰富的语境中，模型比人类更保守；而在较短、事实导向的语境中，人类反而更保守。代码已开源。
原文摘要 · Abstract (English)
The interpretation of implicit meanings is an integral aspect of human communication. However, this framework may not transfer to interactions with Large Language Models (LLMs). To investigate this, we introduce the task of Implicit Information Extraction (IIE) and propose an LLM-based IIE pipeline that builds a structured knowledge graph from a context sentence by extracting relational triplets, validating implicit inferences, and analyzing temporal relations. We evaluate two LLMs against crowdsourced human judgments on two datasets. We find that humans agree with most model triplets yet consistently propose many additions, indicating limited coverage in current LLM-based IIE. Moreover, in our experiments, models appear to be more conservative about implicit inferences than humans in socially rich contexts, whereas humans become more conservative in shorter, fact-oriented contexts. Our code is available at https://github.com/Antonio-Dee/IIE_from_LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。