arXiv:2501.05478cs.CLcs.AI2025-01中稿 · AAAI被引 3

首次评估多语言模型在机器人导航中的表现,验证阿拉伯语指令下的推理能力。

Language and Planning in Robotic Navigation: A Multilingual Evaluation of State-of-the-Art Models

  • 基于NavGPT框架,用纯语言模型实现零样本导航动作预测。
  • 英语下模型表现优异,但阿拉伯语中部分模型因解析问题表现不佳。
  • 揭示语言模型在导航规划中的局限性,适合关注多语言AI应用的研究者。

大型语言模型(LLMs)如GPT-4,在涵盖多个领域的海量数据上训练后,展现出显著的推理、理解与规划能力。本研究首次在机器人视觉-语言导航(VLN)领域整合阿拉伯语,填补了该方向长期研究空白。我们对包括GPT-4o mini、Llama 3 8B、Phi-3 medium 14B及专为阿拉伯语设计的Jais在内的多语言小模型进行了全面评估。采用基于纯语言模型的NavGPT框架,通过R2R数据集进行零样本序列动作预测,检验语言对导航推理的影响。实验表明,模型在英文指令下具备高水平规划能力;但在阿拉伯语指令下,部分模型因自身能力局限、性能欠佳及解析问题导致推理失败。结果强调提升语言模型在导航中的规划与推理能力的重要性,指出这是未来发展的关键方向,同时释放阿拉伯语模型在真实场景中的应用潜力。

原文摘要 · Abstract (English)

Large Language Models (LLMs) such as GPT-4, trained on huge amount of datasets spanning multiple domains, exhibit significant reasoning, understanding, and planning capabilities across various tasks. This study presents the first-ever work in Arabic language integration within the Vision-and-Language Navigation (VLN) domain in robotics, an area that has been notably underexplored in existing research. We perform a comprehensive evaluation of state-of-the-art multi-lingual Small Language Models (SLMs), including GPT-4o mini, Llama 3 8B, and Phi-3 medium 14B, alongside the Arabic-centric LLM, Jais. Our approach utilizes the NavGPT framework, a pure LLM-based instruction-following navigation agent, to assess the impact of language on navigation reasoning through zero-shot sequential action prediction using the R2R dataset. Through comprehensive experiments, we demonstrate that our framework is capable of high-level planning for navigation tasks when provided with instructions in both English and Arabic. However, certain models struggled with reasoning and planning in the Arabic language due to inherent limitations in their capabilities, sub-optimal performance, and parsing issues. These findings highlight the importance of enhancing planning and reasoning capabilities in language models for effective navigation, emphasizing this as a key area for further development while also unlocking the potential of Arabic-language models for impactful real-world applications.

机器人导航多语言模型语言推理阿拉伯语

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。