arXiv:2506.10172cs.ROcs.AI2025-06被引 1

用大模型理解指令,轻量规划实现快速导航

A Navigation Framework Utilizing Vision-Language Models

  • 分拆视觉语言理解与动作规划,用冻结大模型降低计算开销
  • 在Room-to-Room基准上取得可比基线性能,支持实时部署
  • 适合希望快速构建高效导航系统的研究者与开发者

视觉语言导航(VLN)是具身人工智能中的复杂挑战,要求智能体理解自然语言指令并在视觉丰富的陌生环境中导航。尽管大型视觉语言模型(如CLIP和Flamingo)显著提升了多模态理解能力,但带来了计算成本高和实时部署难的问题。本文提出一种模块化、即插即用的导航框架,将视觉语言理解与动作规划解耦。通过集成冻结的视觉语言模型Qwen2.5-VL-7B-Instruct与轻量级规划逻辑,实现无需大量微调的灵活、快速、可适应导航。框架采用提示工程、结构化历史管理及双帧视觉输入策略,增强跨步骤决策连续性。我们在VLN-CE设置下的Room-to-Room基准上,使用Matterport3D数据集与Habitat-Lab仿真环境进行评估。尽管在严格设定下对未见环境的泛化能力仍有不足,但该模块化方法为可扩展、高效的导航系统奠定了基础,未来可通过引入更强环境先验与多模态输入融合进一步提升性能。

原文摘要 · Abstract (English)

Vision-and-Language Navigation (VLN) presents a complex challenge in embodied AI, requiring agents to interpret natural language instructions and navigate through visually rich, unfamiliar environments. Recent advances in large vision-language models (LVLMs), such as CLIP and Flamingo, have significantly improved multimodal understanding but introduced new challenges related to computational cost and real-time deployment. In this project, we propose a modular, plug-and-play navigation framework that decouples vision-language understanding from action planning. By integrating a frozen vision-language model, Qwen2.5-VL-7B-Instruct, with lightweight planning logic, we aim to achieve flexible, fast, and adaptable navigation without extensive model fine-tuning. Our framework leverages prompt engineering, structured history management, and a two-frame visual input strategy to enhance decision-making continuity across navigation steps. We evaluate our system on the Room-to-Room benchmark within the VLN-CE setting using the Matterport3D dataset and Habitat-Lab simulation environment. Although our initial results reveal challenges in generalizing to unseen environments under strict evaluation settings, our modular approach lays a foundation for scalable and efficient navigation systems, highlighting promising directions for future improvement through enhanced environmental priors and expanded multimodal input integration.

视觉语言导航大模型应用具身智能轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。