用手绘草图+深度图生成无人机3D飞行路径,无需训练即可用在真实环境。
SketchPlan: Diffusion Based Drone Planning From Human Sketches
- 用扩散模型从手绘草图和深度图推断3D飞行轨迹。
- 真实场景测试中低/中等障碍物环境成功率100%,高障碍物环境40%。
- 结合人工标注与自动生成数据,提升对人类意图的理解能力。
我们提出SketchPlan,一种基于扩散模型的无人机规划方法,通过解析深度图上的2D手绘草图生成3D飞行路径。系统包含两个组件:SketchAdapter将人类草图映射为投影后的2D路径,DiffPath扩散模型则从2D投影和第一视角深度图推断3D轨迹。模型实现零样本模拟到现实迁移,在未见过的真实环境中生成准确且安全的飞行路径。为训练模型,我们构建了32,000条飞行路径的合成数据集,使用多样化的照片级3D高斯泼溅场景,并自动计算3D路径在相机平面上的2D投影作为标签。由于真实手绘草图与理想投影差异显著,我们额外标注了872条3D路径的人类草图,用于训练SketchAdapter以从草图推断2D投影。实验表明,结合人工与自动生成数据、模块化设计显著提升了模型理解人类意图和推断3D路径的能力。真实无人机测试中,低/中等遮挡环境下成功率100%,未知高遮挡环境下达40%,优于关键对比方法20%-60%的任务完成率。
原文摘要 · Abstract (English)
We propose SketchPlan, a diffusion-based planner that interprets 2D hand-drawn sketches over depth images to generate 3D flight paths for drone navigation. SketchPlan comprises two components: a SketchAdapter that learns to map the human sketches to projected 2D paths, and DiffPath, a diffusion model that infers 3D trajectories from 2D projections and a first person view depth image. Our model achieves zero-shot sim-to-real transfer, generating accurate and safe flight paths in previously unseen real-world environments. To train the model, we build a synthetic dataset of 32k flight paths using a diverse set of photorealistic 3D Gaussian Splatting scenes. We automatically label the data by computing 2D projections of the 3D flight paths onto the camera plane, and use this to train the DiffPath diffusion model. However, since real human 2D sketches differ significantly from ideal 2D projections, we additionally label 872 of the 3D flight paths with real human sketches and use this to train the SketchAdapter to infer the 2D projection from the human sketch. We demonstrate SketchPlan's effectiveness in both simulated and real-world experiments, and show through ablations that training on a mix of human labeled and auto-labeled data together with a modular design significantly boosts its capabilities to correctly interpret human intent and infer 3D paths. In real-world drone tests, SketchPlan achieved 100\% success in low/medium clutter and 40\% in unseen high-clutter environments, outperforming key ablations by 20-60\% in task completion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。