评测大模型在规划任务中的推理能力，发现其表现仍有显著差距。
ACPBench: Reasoning about Action, Change, and Planning
- 基于形式化语言构建13个规划领域，自动生成带正确解的推理题。
- 22个大模型测试显示，OpenAI o1在选择题上提升明显，但布尔题无进展。
- 适合关注大模型规划与多步推理能力的研究者和开发者使用。
越来越多研究将大语言模型（LLMs）作为代理，在需要规划与多步推理的领域中协调工作流并做出决策。因此，评估LLMs在规划核心能力上的表现至关重要。本文提出ACPBench，一个用于评估规划领域推理任务的基准。该基准包含13个规划领域上的7类推理任务，数据来自用形式化语言描述的规划领域，可自动生成具有可证明正确解的问题，实现大规模扩展而无需额外人工投入。对22个LLMs及OpenAI o1推理模型的全面评估揭示了模型推理能力的巨大差距。结果显示，OpenAI o1在多项选择题上性能显著提升，但在布尔问题上未见明显进步。ACPBench数据集已开源：https://ibm.github.io/ACPBench。
原文摘要 · Abstract (English)
There is an increasing body of work using Large Language Models (LLMs) as agents for orchestrating workflows and making decisions in domains that require planning and multi-step reasoning. As a result, it is imperative to evaluate LLMs on core skills required for planning. In this work, we present ACPBench, a benchmark for evaluating the reasoning tasks in the field of planning. The benchmark consists of 7 reasoning tasks over 13 planning domains. The collection is constructed from planning domains described in a formal language. This allows us to synthesize problems with provably correct solutions across many tasks and domains. Further, it allows us the luxury of scale without additional human effort, i.e., many additional problems can be created automatically. Our extensive evaluation of 22 LLMs and OpenAI o1 reasoning models highlights the significant gap in the reasoning capability of the LLMs. Our findings with OpenAI o1, a multi-turn reasoning model, reveal significant gains in performance on multiple-choice questions, yet surprisingly, no notable progress is made on boolean questions. The ACPBench collection is available at https://ibm.github.io/ACPBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。