大模型无需训练即可识别逻辑谬误，效果优于传统方法。
Are LLMs Good Zero-Shot Fallacy Classifiers?
- 用单轮和多轮提示激发大模型的推理能力
- 零样本性能媲美全量标注数据训练的模型
- 特别适合小模型在开放域任务中使用
谬误是推理错误的论证，识别与分类谬误是防止虚假信息、操纵性言论和偏见决策的关键NLP任务。然而，现有谬误分类器受限于充足的标注数据需求，影响其在分布外（OOD）场景下的泛化能力。本文聚焦利用大语言模型（LLMs）进行零样本谬误分类。为激发LLMs的谬误相关知识与推理能力，我们提出多样化的单轮与多轮提示策略，采用提取、摘要和思维链等任务特定指令。在基准数据集上的全面实验表明，LLMs在单轮提示下已实现可接受的零样本表现，优于最佳全样本基线模型，在所有分布外推理场景及部分开放域任务中表现更优。新颖的多轮提示策略进一步提升性能，尤其对小规模模型效果显著。分析揭示了零样本谬误分类的未来研究方向。代码与数据详见：https://github.com/panFJCharlotte98/Fallacy_Detection。
原文摘要 · Abstract (English)
Fallacies are defective arguments with faulty reasoning. Detecting and classifying them is a crucial NLP task to prevent misinformation, manipulative claims, and biased decisions. However, existing fallacy classifiers are limited by the requirement for sufficient labeled data for training, which hinders their out-of-distribution (OOD) generalization abilities. In this paper, we focus on leveraging Large Language Models (LLMs) for zero-shot fallacy classification. To elicit fallacy-related knowledge and reasoning abilities of LLMs, we propose diverse single-round and multi-round prompting schemes, applying different task-specific instructions such as extraction, summarization, and Chain-of-Thought reasoning. With comprehensive experiments on benchmark datasets, we suggest that LLMs could be potential zero-shot fallacy classifiers. In general, LLMs under single-round prompting schemes have achieved acceptable zero-shot performances compared to the best full-shot baselines and can outperform them in all OOD inference scenarios and some open-domain tasks. Our novel multi-round prompting schemes can effectively bring about more improvements, especially for small LLMs. Our analysis further underlines the future research on zero-shot fallacy classification. Codes and data are available at: https://github.com/panFJCharlotte98/Fallacy_Detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。