测试大模型在带时间约束的自然语言规划任务中的表现。
LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language
- 将已有规划环境添加时间约束，转为自然语言任务
- 主流大模型在高约束任务中表现明显下降
- 可扩展性强，适合未来评估更复杂规划能力
由于具备推理能力，大语言模型（LLMs）已被用于评估自然语言描述的规划任务。然而，现有测试大多未包含约束条件。为在真实场景中部署，尤其需确保安全等约束的遵守，必须评估模型在有约束规划任务中的表现。我们提出 LexiCon——一个基于自然语言的约束规划基准，包含一系列可评估 LLM 规划能力的环境。其核心思想是将现有规划环境施加时间约束，并转化为自然语言任务给 LLM 求解。LexiCon 具有可扩展性：可通过新增无约束环境生成器自动构建时间约束，使基准持续适应模型进步。实验表明，包括 GPT-5、o3、R1 等在内的先进推理模型，其性能随规划任务约束程度增加而显著下降。
原文摘要 · Abstract (English)
Owing to their reasoning capabilities, large language models (LLMs) have been evaluated on planning tasks described in natural language. However, LLMs have largely been tested on planning domains without constraints. In order to deploy them in real-world settings where adherence to constraints, in particular safety constraints, is critical, we need to evaluate their performance on constrained planning tasks. We introduce LexiCon -- a natural language-based (Lexi) constrained (Con) planning benchmark, consisting of a suite of environments, that can be used to evaluate the planning capabilities of LLMs in a principled fashion. The core idea behind LexiCon is to take existing planning environments and impose temporal constraints on the states. These constrained problems are then translated into natural language and given to an LLM to solve. A key feature of LexiCon is its extensibility. That is, the set of supported environments can be extended with new (unconstrained) environment generators, for which temporal constraints are constructed automatically. This renders LexiCon future-proof: the hardness of the generated planning problems can be increased as the planning capabilities of LLMs improve. Our experiments reveal that the performance of state-of-the-art LLMs, including reasoning models like GPT-5, o3, and R1, deteriorates as the degree of constrainedness of the planning tasks increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。