用2D游戏地图测试大模型的规划能力，发现现有模型仍显不足。
GameTraversalBenchmark: Evaluating Planning Abilities Of Large Language Models Through Traversing 2D Game Maps
- 设计游戏地图基准测试，评估模型规划路径能力。
- GPT-4-Turbo得分44.97%，o1模型达67.84%但仍有提升空间。
- 适合研究大模型推理与规划能力的研究者参考。
大型语言模型（LLMs）在自然语言生成与理解方面已取得显著进展。然而，它们在非语言领域的规划能力仍不明确。本文提出GameTraversalBenchmark（GTB），一个包含多种2D网格游戏地图的基准测试，评估模型在最少步数和最少生成错误下完成目标路径的能力。评估结果显示，GPT-4-Turbo在综合得分（GTBS）上达到最高44.97%；初步测试的大规模推理模型o1在该指标上达67.84%，表明当前模型仍面临挑战。代码、数据及文档已开源于https://github.com/umair-nasir14/Game-Traversal-Benchmark。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently demonstrated great success in generating and understanding natural language. While they have also shown potential beyond the domain of natural language, it remains an open question as to what extent and in which way these LLMs can plan. We investigate their planning capabilities by proposing GameTraversalBenchmark (GTB), a benchmark consisting of diverse 2D grid-based game maps. An LLM succeeds if it can traverse through given objectives, with a minimum number of steps and a minimum number of generation errors. We evaluate a number of LLMs on GTB and found that GPT-4-Turbo achieved the highest score of 44.97% on GTB\_Score (GTBS), a composite score that combines the three above criteria. Furthermore, we preliminarily test large reasoning models, namely o1, which scores $67.84\%$ on GTBS, indicating that the benchmark remains challenging for current models. Code, data, and documentation are available at https://github.com/umair-nasir14/Game-Traversal-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。