用语篇连接词反推模型对新实体的认知能力
WUGNECTIVES: Novel Entity Inferences of Language Models from Discourse Connectives
- 设计新数据集，通过连接词推断模型对新实体属性的理解
- 微调后模型在多数连接词上表现显著提升，但让步类连接词仍困难
- 适合研究语言模型推理机制与语篇线索认知功能的学者
世界知识在预测两个论点间语篇关系所用连接词方面至关重要，语言模型（LMs）在此任务中普遍表现良好。本文反其道而行之，研究逆问题：语篇连接词能否帮助语言模型理解世界？为此，我们构建了 WUGNECTIVES，一个包含 8,880 条刺激样本的数据集，评估语言模型在连接词将实体关联到特定属性的情境下，对新实体的推理能力。我们测试了 17 种不同规模和训练方式的语言模型，发现使模型具备推理行为的微调可显著提升多数连接词上的表现。然而，模型在表达让步意义的连接词上整体表现较差，存在显著差异。研究结果为深入探究语言线索在语言模型中所起的功能作用提供了新路径。数据集已开源：https://github.com/kanishkamisra/wugnectives
原文摘要 · Abstract (English)
The role of world knowledge has been particularly crucial to predict the discourse connective that marks the discourse relation between two arguments, with language models (LMs) being generally successful at this task. We flip this premise in our work, and instead study the inverse problem of understanding whether discourse connectives can inform LMs about the world. To this end, we present WUGNECTIVES, a dataset of 8,880 stimuli that evaluates LMs' inferences about novel entities in contexts where connectives link the entities to particular attributes. On investigating 17 different LMs at various scales, and training regimens, we found that tuning an LM to show reasoning behavior yields noteworthy improvements on most connectives. At the same time, there was a large variation in LMs' overall performance across connective type, with all models systematically struggling on connectives that express a concessive meaning. Our findings pave the way for more nuanced investigations into the functional role of language cues as captured by LMs. We release WUGNECTIVES at https://github.com/kanishkamisra/wugnectives
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。