测试大模型解决逻辑编程问题的能力，发现其在核心任务上表现不佳。
Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version)
- 构建了专用于ASP的基准测试集ASPBench，包含三类任务。
- 14个主流大模型在简单任务中表现尚可，但计算答案集时严重失效。
- 研究揭示大模型缺乏符号推理能力，适合对逻辑推理有需求的研究者。
答案集编程（ASP）是一种强大的非单调推理范式。近期大型语言模型（LLMs）在逻辑推理方面展现出潜力。然而，当前对LLMs在ASP领域能力的评估仍显不足：多数研究使用过于简化的ASP程序，不支持否定、析取或多个答案集。此外，缺乏针对ASP求解设计的任务基准。为弥补这一空白，我们提出了ASPBench，一个全面的ASP基准测试集，包含三类特定任务：ASP蕴含判断、答案集验证和答案集计算。对ASPBench的广泛评估显示，尽管14个前沿大模型（包括deepseek-r1、o4-mini、gemini-2.5-flash-thinking）在前两项较简单任务中表现良好，但在核心的答集计算任务上表现不佳。该结果揭示了当前大模型在ASP求解中的局限性，凸显了需更有效地融合符号推理的新方法。代码与数据集已开源：https://github.com/HomuraT/ASPBench。
原文摘要 · Abstract (English)
Answer Set Programming (ASP) is a powerful paradigm for non-monotonic reasoning. Recently, large language models (LLMs) have demonstrated promising capabilities in logical reasoning. Despite this potential, current evaluations of LLM capabilities in ASP are often limited. Existing works normally employ overly simplified ASP programs, do not support negation, disjunction, or multiple answer sets. Furthermore, there is a lack of benchmarks that introduce tasks specifically designed for ASP solving. To bridge this gap, we introduce ASPBench, a comprehensive ASP benchmark, including three ASP specific tasks: ASP entailment, answer set verification, and answer set computation. Our extensive evaluations on ASPBench reveal that while 14 state-of-the-art LLMs, including \emph{deepseek-r1}, \emph{o4-mini}, and \emph{gemini-2.5-flash-thinking}, perform relatively well on the first two simpler tasks, they struggle with answer set computation, which is the core of ASP solving. These findings offer insights into the current limitations of LLMs in ASP solving. This highlights the need for new approaches that integrate symbolic reasoning capabilities more effectively. The code and dataset are available at https://github.com/HomuraT/ASPBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。