让无人机听懂人话,自动抓物递给人,实现自然交互。
DroneVLA: VLA based Aerial Manipulation
- 用视觉语言动作模型理解自然语言指令,生成抓取任务队列。
- 实测定位误差最大0.164米,均方根误差0.084米,精度可靠。
- 结合人体姿态估计,实现安全舒适的无人机递物交接。
随着飞行平台从被动观察者演变为主动操作者,挑战转向设计直观接口,使非专业用户能自然地操控系统。本文提出一种基于视觉-语言-动作(VLA)模型的自主空中操作新方案,可解析高层自然语言指令,完成物体检索并交付给用户。系统集成基于MediaPipe的Grounding DINO与VLA模型,搭配自研配备1-自由度夹持器和Intel RealSense RGB-D相机的无人机。VLA进行语义推理,生成场景中相关物体的优先级抓取任务队列;利用Grounding DINO与动态A*规划算法实现导航与安全移动。为确保交接阶段的安全与自然交互,系统采用由MediaPipe驱动的人体中心控制器,实时估算人体姿态,支持视觉伺服控制,使无人机始终稳定位于用户正前方,便于舒适递送。真实世界实验验证了系统的有效性,定位误差最大值为0.164米,平均欧氏距离误差0.070米,均方根误差0.084米,证明了VLA在空中操作中的可行性。
原文摘要 · Abstract (English)
As aerial platforms evolve from passive observers to active manipulators, the challenge shifts toward designing intuitive interfaces that allow non-expert users to command these systems naturally. This work introduces a novel concept of autonomous aerial manipulation system capable of interpreting high-level natural language commands to retrieve objects and deliver them to a human user. The system is intended to integrate a MediaPipe based on Grounding DINO and a Vision-Language-Action (VLA) model with a custom-built drone equipped with a 1-DOF gripper and an Intel RealSense RGB-D camera. VLA performs semantic reasoning to interpret the intent of a user prompt and generates a prioritized task queue for grasping of relevant objects in the scene. Grounding DINO and dynamic A* planning algorithm are used to navigate and safely relocate the object. To ensure safe and natural interaction during the handover phase, the system employs a human-centric controller driven by MediaPipe. This module provides real-time human pose estimation, allowing the drone to employ visual servoing to maintain a stable, distinct position directly in front of the user, facilitating a comfortable handover. We demonstrate the system's efficacy through real-world experiments for localization and navigation, which resulted in a 0.164m, 0.070m, and 0.084m of max, mean euclidean, and root-mean squared errors, respectively, highlighting the feasibility of VLA for aerial manipulation operations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。