用大模型让无人机自动扫描未知环境中的3D结构,只需简单指令即可完成。
FlyCo: Foundation Model-Empowered Drones for Autonomous 3D Structure Scanning in Open-World Environments
- 通过感知-预测-规划闭环,将文本或标注指令转为自适应飞行路径。
- 在真实与仿真环境中实现高精度、实时安全的全覆盖扫描,效率优于现有方法。
- 适合需要低人工干预的复杂场景三维重建,如灾害勘查、工业巡检。
尽管广泛应用,无人机在开放世界中自主扫描目标结构仍具挑战性。现有方法依赖严格假设或大量人工先验,限制了实用性、效率与适应性。近年来的大模型(FM)为此提供了新可能。本文探究关键问题:如何有效集成大模型知识以实现该任务?提出FlyCo,一种基于大模型的感知-预测-规划闭环系统,支持在多样未知开放环境中完全自主、提示驱动的3D目标扫描。FlyCo将低投入的人类提示(文本、视觉标注)直接转化为精确自适应的扫描飞行路径,包含三阶段:(1) 感知融合流式传感器数据与视觉语言大模型,实现鲁棒的目标定位与跟踪;(2) 预测利用大模型知识并结合多模态线索,推断部分观测下目标的完整几何;(3) 规划基于预测前瞻生成高效且安全的飞行路径,确保全面覆盖。进一步设计关键组件以提升开放世界目标定位效率与鲁棒性,增强形状精度、零样本泛化能力与时间稳定性,并平衡长时飞行效率与实时计算及在线避障。大量真实与仿真实验表明,FlyCo实现了高精度场景理解、高效率与实时安全性,优于现有范式,且人类参与更少,验证了架构的实用性。全面消融实验确认各组件贡献。FlyCo亦可作为灵活可扩展的蓝图,持续利用未来大模型与机器人进展。代码将开源。
原文摘要 · Abstract (English)
Autonomous 3D scanning of open-world target structures via drones remains challenging despite broad applications. Existing paradigms rely on restrictive assumptions or effortful human priors, limiting practicality, efficiency, and adaptability. Recent foundation models (FMs) offer great potential to bridge this gap. This paper investigates a critical research problem: What system architecture can effectively integrate FM knowledge for this task? We answer it with FlyCo, a principled FM-empowered perception-prediction-planning loop enabling fully autonomous, prompt-driven 3D target scanning in diverse unknown open-world environments. FlyCo directly translates low-effort human prompts (text, visual annotations) into precise adaptive scanning flights via three coordinated stages: (1) perception fuses streaming sensor data with vision-language FMs for robust target grounding and tracking; (2) prediction distills FM knowledge and combines multi-modal cues to infer the partially observed target's complete geometry; (3) planning leverages predictive foresight to generate efficient and safe paths with comprehensive target coverage. Building on this, we further design key components to boost open-world target grounding efficiency and robustness, enhance prediction quality in terms of shape accuracy, zero-shot generalization, and temporal stability, and balance long-horizon flight efficiency with real-time computability and online collision avoidance. Extensive challenging real-world and simulation experiments show FlyCo delivers precise scene understanding, high efficiency, and real-time safety, outperforming existing paradigms with lower human effort and verifying the proposed architecture's practicality. Comprehensive ablations validate each component's contribution. FlyCo also serves as a flexible, extensible blueprint, readily leveraging future FM and robotics advances. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。