arXiv:2411.04006cs.ROcs.AI2024-11被引 7

用现成视觉语言模型实现无需训练的机器人规划,靠提问和记忆检索搞定。

Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval

  • 通过视觉问答和上下文学习,不需微调直接用预训练模型做规划。
  • 在第三人称视角下让基线模型导航能力提升约50%,仅需20次演示。
  • 适合想快速部署、无额外传感器需求的机器人研发团队。

本研究探索了现成视觉语言模型(VLMs)在自主导航场景中用于高层机器人规划的潜力。现有基于学习的路径规划方法通常需要大量任务专属训练或微调,而本文展示如何在大多数实际情况下避免此类训练。为此,提出Select2Plan(S2P)——一种完全无需微调或专门训练的新型无训练框架。该方法利用结构化视觉问答(VQA)与上下文学习(ICL),大幅减少数据采集需求,仅需传统训练模型所用任务数据的一小部分,甚至可仅依赖在线数据。该方法以灵活高效的方式利用通用训练的VLM,且除简单单目摄像头外无需额外传感设备。我们在多种场景类型、上下文来源和感知配置下验证了其适应性。评估涵盖两种不同场景:传统第一人称视角(FPV)与基础设施驱动的第三人称视角(TPV)导航。结果表明,在TPV场景中,本方法使基线VLM导航能力显著提升约50%;在FPV场景中表现接近训练模型,且仅需20次示范。

原文摘要 · Abstract (English)

This study explores the potential of off-the-shelf Vision-Language Models (VLMs) for high-level robot planning in the context of autonomous navigation. Indeed, while most of existing learning-based approaches for path planning require extensive task-specific training/fine-tuning, we demonstrate how such training can be avoided for most practical cases. To do this, we introduce Select2Plan (S2P), a novel training-free framework for high-level robot planning which completely eliminates the need for fine-tuning or specialised training. By leveraging structured Visual Question-Answering (VQA) and In-Context Learning (ICL), our approach drastically reduces the need for data collection, requiring a fraction of the task-specific data typically used by trained models, or even relying only on online data. Our method facilitates the effective use of a generally trained VLM in a flexible and cost-efficient way, and does not require additional sensing except for a simple monocular camera. We demonstrate its adaptability across various scene types, context sources, and sensing setups. We evaluate our approach in two distinct scenarios: traditional First-Person View (FPV) and infrastructure-driven Third-Person View (TPV) navigation, demonstrating the flexibility and simplicity of our method. Our technique significantly enhances the navigational capabilities of a baseline VLM of approximately 50% in TPV scenario, and is comparable to trained models in the FPV one, with as few as 20 demonstrations.

机器人规划视觉语言模型零样本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。