测试大模型在逻辑推理中的鲁棒性，发现对抗噪声和反事实语句会显著影响推理结果。
Investigating the Robustness of Deductive Reasoning with Large Language Models
- 设计两类扰动：对抗噪声与反事实陈述，生成七组测试数据。
- 自动形式化方法受噪声影响大，所有方法都易受反事实干扰。
- 详细反馈虽减少语法错误，但整体准确率未提升，说明纠错能力仍弱。
大型语言模型（LLMs）在多项基于推理的自然语言处理任务中表现优异，暗示其具备一定的逻辑推理能力。然而，现有研究尚不清楚这些模型在非形式化与自动形式化方法下进行逻辑推理时的鲁棒性如何。此外，尽管已有大量基于LLM的推理方法被提出，但缺乏对其设计组件影响的系统性分析。为解决这两个问题，本文首次对形式化与非形式化的基于LLM的逻辑推理方法的鲁棒性进行了研究。我们构建了一个框架，包含两类扰动：对抗性噪声与反事实陈述，联合生成七组扰动数据集。根据推理格式、形式化语法及错误恢复反馈，梳理了当前LLM推理者的布局。实验结果表明，对抗性噪声主要影响自动形式化方法，而反事实陈述则影响所有方法。尽管详细反馈能减少语法错误，但整体准确率未提升，反映出基于LLM的方法在自我纠正方面仍存在挑战。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have been shown to achieve impressive results for many reasoning-based NLP tasks, suggesting a degree of deductive reasoning capability. However, it remains unclear to which extent LLMs, in both informal and autoformalisation methods, are robust on logical deduction tasks. Moreover, while many LLM-based deduction methods have been proposed, a systematic study that analyses the impact of their design components is lacking. Addressing these two challenges, we propose the first study of the robustness of formal and informal LLM-based deductive reasoning methods. We devise a framework with two families of perturbations: adversarial noise and counterfactual statements, which jointly generate seven perturbed datasets. We organize the landscape of LLM reasoners according to their reasoning format, formalisation syntax, and feedback for error recovery. The results show that adversarial noise affects autoformalisation, while counterfactual statements influence all approaches. Detailed feedback does not improve overall accuracy despite reducing syntax errors, pointing to the challenge of LLM-based methods to self-correct effectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。