用逻辑编程提升大模型的空间推理能力,效果显著。
Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMs
- 分模块设计让大模型与逻辑程序协同工作。
- 在两个数据集上准确率分别达82%和69%,提升40%-50%。
- 适合需要可解释推理的AI系统开发者。
大语言模型在多项任务中表现优异,但在空间推理方面仍存在不足。本文提出一种新型神经符号框架,通过大模型与答案集编程(ASP)之间的迭代反馈,增强其空间推理能力。我们在StepGame和SparQA两个基准数据集上评估了三种策略:(1)直接提示基线,(2)事实+规则提示,(3)基于DSPy的大模型+ASP管道并带迭代优化。实验结果表明,该管道显著优于基线方法,在StepGame上平均准确率达82%,在SparQA上达69%,分别比直接提示提升40-50%和8-15%。成功源于三项关键创新:(1)通过模块化设计分离语义解析与逻辑推理;(2)大模型与ASP求解器间的迭代反馈机制提升了程序生成率;(3)具备鲁棒错误处理能力,能应对解析、对齐和求解失败。此外,我们提出的Facts+Rules作为轻量替代方案,在复杂SparQA数据集上表现接近,同时降低计算开销。对不同大模型架构(Deepseek、Llama3-70B、GPT-4.0 mini)的分析显示该框架具备通用性,并揭示了实现复杂度与推理能力间的权衡,有助于构建更可解释、可靠的AI系统。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks, yet they often struggle with spatial reasoning. This paper presents a novel neural-symbolic framework that enhances LLMs' spatial reasoning abilities through iterative feedback between LLMs and Answer Set Programming (ASP). We evaluate our approach on two benchmark datasets: StepGame and SparQA, implementing three distinct strategies: (1) direct prompting baseline, (2) Facts+Rules prompting, and (3) DSPy-based LLM+ASP pipeline with iterative refinement. Our experimental results demonstrate that the LLM+ASP pipeline significantly outperforms baseline methods, achieving an average 82% accuracy on StepGame and 69% on SparQA, marking improvements of 40-50% and 8-15% respectively over direct prompting. The success stems from three key innovations: (1) effective separation of semantic parsing and logical reasoning through a modular pipeline, (2) iterative feedback mechanism between LLMs and ASP solvers that improves program rate, and (3) robust error handling that addresses parsing, grounding, and solving failures. Additionally, we propose Facts+Rules as a lightweight alternative that achieves comparable performance on complex SparQA dataset, while reducing computational overhead.Our analysis across different LLM architectures (Deepseek, Llama3-70B, GPT-4.0 mini) demonstrates the framework's generalizability and provides insights into the trade-offs between implementation complexity and reasoning capability, contributing to the development of more interpretable and reliable AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。