让大模型学会走迷宫，准确率从0提升到93%。
AlphaMaze: Enhancing Large Language Models' Spatial Intelligence via GRPO
- 先用标注迷宫数据微调，再用新算法优化决策链。
- 在合成迷宫上，准确率从86%提升至93%。
- 适合研究视觉推理、机器人导航的团队参考。
大型语言模型在语言处理上表现优异，但在需要真实视觉空间推理的任务中常表现不佳。本文提出一种两阶段训练框架，使标准LLM具备迷宫导航的视觉推理能力。首先，在精心构建的迷宫标记数据集上使用监督微调（SFT），教会模型预测逐步移动指令；随后，采用类似DeepSeekR1的分组相对策略优化（GRPO）技术，结合精心设计的奖励函数，优化模型的序列决策并激发涌现式思维链行为。在合成生成的迷宫上的实验表明，基线模型无法完成导航，SFT微调后准确率达86%，进一步经GRPO优化后提升至93%。定性分析显示，GRPO增强了模型的鲁棒性和自我修正能力，凸显了该方法在弥合语言模型与视觉空间任务差距方面的潜力。研究成果对机器人、自主导航等需融合视觉与序列推理的领域具有重要意义。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive capabilities in language processing, yet they often struggle with tasks requiring genuine visual spatial reasoning. In this paper, we introduce a novel two-stage training framework designed to equip standard LLMs with visual reasoning abilities for maze navigation. First, we leverage Supervised Fine Tuning (SFT) on a curated dataset of tokenized maze representations to teach the model to predict step-by-step movement commands. Next, we apply Group Relative Policy Optimization (GRPO)-a technique used in DeepSeekR1-with a carefully crafted reward function to refine the model's sequential decision-making and encourage emergent chain-of-thought behaviors. Experimental results on synthetically generated mazes show that while a baseline model fails to navigate the maze, the SFT-trained model achieves 86% accuracy, and further GRPO fine-tuning boosts accuracy to 93%. Qualitative analyses reveal that GRPO fosters more robust and self-corrective reasoning, highlighting the potential of our approach to bridge the gap between language models and visual spatial tasks. These findings offer promising implications for applications in robotics, autonomous navigation, and other domains that require integrated visual and sequential reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。