构建自然语言转逻辑程序的基准，评估系统在复杂推理上的表现
ASP-Bench: From Natural Language to Logic Programs
- 设计128个自然语言问题，涵盖选择、聚合、优化等ASP特性
- 基于ReAct框架的智能体实现全题解，验证反馈迭代的有效性
- 从7个维度分析建模难度，适用于神经符号系统研究者
自动化将自然语言规范转化为逻辑程序是神经符号工程中的挑战性任务。我们提出ASP-Bench，一个包含128个自然语言问题实例的基准，其中64个基础问题配有简单和困难变体。该基准用于评估将自然语言问题转化为答案集程序（Answer Set Programs, ASP）的系统性能，覆盖了选择规则、聚合和优化等关键特征。每个问题均配备参考验证器，以检查解是否满足原始规格。我们从七个相对独立的推理维度（优化、时间推理、默认逻辑、资源分配、递归、空间推理和数量复杂度）对问题进行表征，提供多维度的建模难度分析。使用基于ReAct（Reason and Act）框架的智能体方法测试该基准，实现了完全饱和求解，表明结合求解器反馈的迭代修正是一种可靠且稳健的自然语言建模方法。通过对多次智能体运行的分析，我们获得了影响问题建模难度的关键因素洞察。
原文摘要 · Abstract (English)
Automating the translation of natural-language specifications into logic programs is a challenging task that affects neurosymbolic engineering. We present ASP-Bench, a benchmark comprising 128 natural language problem instances, 64 base problems with easy and hard variants. It evaluates systems that translate natural-language problems into Answer Set Programs (ASPs), a prominent form of logic programming. It provides systematic coverage of ASP features, including choice rules, aggregates, and optimization. Each problem includes reference validators that check whether solutions satisfy the problem specification. We characterize problems along seven largely independent reasoning aspects (optimization, temporal reasoning, default logic, resource allocation, recursion, spatial reasoning, and quantitative complexity), providing a multidimensional view of modeling difficulty. We test the benchmark using an agentic approach based on the ReAct (Reason and Act) framework, which achieves full saturation, demonstrating that feedback-driven iterative refinement with solver feedback provides a reliable and robust approach for modeling natural language in ASP. Our analysis across multiple agent runs enables us to gain insights into what determines a problem's modeling hardness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。