测试大模型在无视觉条件下的空间推理能力，发现其导航表现严重依赖语言和训练数据。
MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Models
- 用坐标与墙距反馈设计纯空间推理任务，避免视觉干扰
- 多数模型在9×9以上迷宫中循环错误，100%失败由重复访问同一格导致
- 英文到冰岛语性能下降3-4级，说明空间能力源于语言模式而非通用认知
随着大语言模型（LLMs）在机器人与具身智能中日益成为自主代理的核心，理解其空间推理能力对确保真实世界部署的可靠性至关重要。尽管语言理解取得进展，现有研究缺乏对无视觉线索下LLM空间导航表现的评估，而这是感官受限场景中代理的基本要求。本文提出MazeEval基准，通过基于坐标的迷宫导航任务，隔离并评估LLMs的纯空间推理能力。方法采用函数调用接口，模型在5×5至15×15网格迷宫中仅凭坐标反馈和距墙距离导航，不使用视觉输入以测试基础空间认知。我们在英语和冰岛语环境下对8个顶尖LLM进行相同迷宫测试。结果揭示显著差异：OpenAI的O3在30×30迷宫中表现完美，其他模型在超过9×9的迷宫中出现灾难性失败，100%的失败归因于反复访问同一单元格（至少10次）。在冰岛语中性能大幅下降，解决的迷宫尺寸比英语小3-4级，表明空间推理能力源自语言模式而非跨语言通用机制。该结果对全球部署基于LLM的自主系统具有重要启示，提示空间智能仍受限于训练数据，亟需架构创新以实现多语言环境下的可靠导航。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) increasingly power autonomous agents in robotics and embodied AI, understanding their spatial reasoning capabilities becomes crucial for ensuring reliable real-world deployment. Despite advances in language understanding, current research lacks evaluation of how LLMs perform spatial navigation without visual cues, a fundamental requirement for agents operating with limited sensory information. This paper addresses this gap by introducing MazeEval, a benchmark designed to isolate and evaluate pure spatial reasoning in LLMs through coordinate-based maze navigation tasks. Our methodology employs a function-calling interface where models navigate mazes of varying complexity ($5\times 5$ to $15\times 15$ grids) using only coordinate feedback and distance-to-wall information, excluding visual input to test fundamental spatial cognition. We evaluate eight state-of-the-art LLMs across identical mazes in both English and Icelandic to assess cross-linguistic transfer of spatial abilities. Our findings reveal striking disparities: while OpenAI's O3 achieves perfect navigation for mazes up to size $30\times 30$, other models exhibit catastrophic failure beyond $9\times 9$ mazes, with 100% of failures attributed to excessive looping behavior where models revisit a cell at least 10 times. We document a significant performance degradation in Icelandic, with models solving mazes 3-4 sizes smaller than in English, suggesting spatial reasoning in LLMs emerges from linguistic patterns rather than language-agnostic mechanisms. These results have important implications for global deployment of LLM-powered autonomous systems, showing spatial intelligence remains fundamentally constrained by training data availability and highlighting the need for architectural innovations to achieve reliable navigation across linguistic contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。