研究大模型在上下文学习中理解否定句的系统性问题
Revisiting the Systematicity in Negation in the Era of In-Context Learning
- 通过上下文示例，模型可部分识别否定表达和作用范围
- 不同输出格式影响否定范围识别效果，未达完美性能
- 否定线索提取可构建稳定函数向量，范围识别更难
理解否定句的含义仍是语言模型的一大挑战，即使在大语言模型（LLMs）时代亦然。本文从行为系统性和表征系统性两个角度分析了LLM对否定的理解。在行为系统性方面，通过示范和上下文学习，LLMs能在一定程度上识别句子中的否定表达及其作用范围，但未能达到理想表现。尤其值得注意的是，模型在否定范围识别上的难度随输出格式变化而波动。在表征系统性方面，我们评估了从上下文示例中构建用于关键否定理解任务的功能向量的稳健性。实验表明，虽然否定线索提取任务可成功构造功能向量，但识别否定作用范围的任务则更具挑战性。
原文摘要 · Abstract (English)
Understanding the meaning of negated sentences remains one of the challenges for language models, even in the era of large language models (LLMs). We analyze systematicity regarding LLM understanding of negation from two perspectives: behavioral systematicity and representational systematicity. For behavioral systematicity, we confirm that through demonstrations and in-context learning, LLMs can recognize negation expressions and scope within sentences to some extent, but they fail to achieve perfect performance. In particular, the difficulty of the negation scope recognition for models varies depending on the output format. For representational systematicity, we analyze the extent to which function vectors can be robustly constructed from in-context examples for tasks that are essential to understanding negation. The experiments suggest that while function vectors can be composed for negation cue extraction tasks, extracting function vectors for recognizing scope is more challenging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。