无需训练的无人机导航框架，能理解任意指令并精准飞行。
See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation
- 将语言指令转为图像上的2D航点，再生成3D动作指令。
- 在仿真中性能超越之前最佳方法63%，真实场景也大幅领先。
- 适用于不同视觉语言模型，适合追求通用导航的开发者。
我们提出See, Point, Fly（SPF），一种基于视觉-语言模型（VLM）的免训练空中视觉与语言导航（AVLN）框架。SPF可基于任意类型的自由文本指令，在任意环境中导航至目标。不同于将动作预测视为文本生成的传统方法，SPF将动作预测视为二维空间定位任务：利用VLM将模糊语言指令分解为输入图像上的一系列2D航点标注，并结合预测的行驶距离，将2D航点转化为3D位移向量作为无人机动作指令。此外，SPF自适应调整行驶距离，提升导航效率。其采用闭环控制方式，支持在动态环境中追踪动态目标。在DRL仿真基准测试中，SPF性能超越此前最优方法63%；在广泛的真实世界评估中亦显著优于强基线。全面消融实验验证了设计有效性。SPF对不同VLM具有优异泛化能力。
原文摘要 · Abstract (English)
We present See, Point, Fly (SPF), a training-free aerial vision-and-language navigation (AVLN) framework built atop vision-language models (VLMs). SPF is capable of navigating to any goal based on any type of free-form instructions in any kind of environment. In contrast to existing VLM-based approaches that treat action prediction as a text generation task, our key insight is to consider action prediction for AVLN as a 2D spatial grounding task. SPF harnesses VLMs to decompose vague language instructions into iterative annotation of 2D waypoints on the input image. Along with the predicted traveling distance, SPF transforms predicted 2D waypoints into 3D displacement vectors as action commands for UAVs. Moreover, SPF also adaptively adjusts the traveling distance to facilitate more efficient navigation. Notably, SPF performs navigation in a closed-loop control manner, enabling UAVs to follow dynamic targets in dynamic environments. SPF sets a new state of the art in DRL simulation benchmark, outperforming the previous best method by an absolute margin of 63%. In extensive real-world evaluations, SPF outperforms strong baselines by a large margin. We also conduct comprehensive ablation studies to highlight the effectiveness of our design choice. Lastly, SPF shows remarkable generalization to different VLMs. Project page: https://spf-web.pages.dev
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。