解析NLI任务的逻辑本质，揭示模型理解推理的真实机制。
Reverse-engineering NLI: A study of the meta-inferential properties of Natural Language Inference
- 通过共享前提和大模型生成数据，检验NLI的元推理一致性
- 发现SNLI数据集实际编码的是非严格蕴含关系而非严格逻辑推导
- 为评估语言模型理解能力提供新视角，适合研究NLU机制者阅读
自然语言推理（NLI）是评估语言模型自然语言理解能力的重要任务，但该任务的逻辑属性长期未被充分理解且常被误读。理解NLI所捕捉的推理概念，对于正确解释模型在该任务上的表现至关重要。本文提出三种对NLI标签集可能的解读，并对它们所蕴含的元推理特性进行系统分析。以SNLI数据集为基础，利用（1）具有相同前提的NLI样本，以及（2）由大语言模型生成的样本，评估在SNLI上训练的模型在元推理一致性上的表现，进而揭示数据集中实际编码的逻辑关系类型。
原文摘要 · Abstract (English)
Natural Language Inference (NLI) has been an important task for evaluating language models for Natural Language Understanding, but the logical properties of the task are poorly understood and often mischaracterized. Understanding the notion of inference captured by NLI is key to interpreting model performance on the task. In this paper we formulate three possible readings of the NLI label set and perform a comprehensive analysis of the meta-inferential properties they entail. Focusing on the SNLI dataset, we exploit (1) NLI items with shared premises and (2) items generated by LLMs to evaluate models trained on SNLI for meta-inferential consistency and derive insights into which reading of the logical relations is encoded by the dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。