用复杂逻辑语言构建数据集，测试大模型的推理能力。
Transformer-based Language Models for Reasoning in the Description Logic ALCQ
- 基于ALCQ逻辑构造38.4万条自然语言推理数据
- 微调后的DeBERTa模型可准确完成蕴含判断
- GPT-3.5仅需9个示例就能显著提升表现
基于Transformer的语言模型在逻辑推理方面取得进展，但现有评估基准大多简单：由少量一阶逻辑语句生成，仅含有限逻辑算子与量词。本文构建了自然语言数据集DELTA$_D$，基于表达力更强的描述逻辑语言$ \mathcal{ALCQ}$，包含384,000个样本，从推理深度和语言复杂度两个维度提升挑战性。我们系统评估了在该数据集上微调的DeBERTa模型及两个大语言模型（GPT-3.5、GPT-4）在少样本提示下的表现。结果表明，经微调的DeBERTa模型能掌握蕴含检查任务；而GPT系列模型即使仅提供9个示例，性能也显著提升。代码与数据集已开源。
原文摘要 · Abstract (English)
Recent advancements in transformer-based language models have sparked research into their logical reasoning capabilities. Most of the benchmarks used to evaluate these models are simple: generated from short (fragments of) first-order logic sentences with only a few logical operators and quantifiers. We construct the natural language dataset, DELTA$_D$, using the expressive description logic language $\mathcal{ALCQ}$. DELTA$_D$ comprises 384K examples and increases in two dimensions: i) reasoning depth, and ii) linguistic complexity. In this way, we systematically investigate the logical reasoning capabilities of a supervised fine-tuned DeBERTa-based model and two large language models (GPT-3.5, GPT-4) with few-shot prompting. We show that the DeBERTa-based model fine-tuned on our dataset can master the entailment checking task. Moreover, the performance of GPTs can improve significantly even when a small number of samples is provided (9 shots). We open-source our code and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。