用高保真虚拟环境训练无人机视觉导航,实测抗干扰能力强。
SOUS VIDE: Cooking Visual Drone Navigation Policies in a Gaussian Splatting Vacuum
- 用高保真高斯泼溅重建场景,快速生成130帧/秒的逼真图像
- 通过专家策略蒸馏出轻量神经网络,20赫兹实时控制飞行
- 实机测试抗30%质量变化、40米/秒阵风等复杂干扰,适合真实场景
我们提出一种名为SOUS VIDE的新模拟器、训练方法和策略架构,用于端到端视觉无人机导航。训练后的策略仅使用机载感知与计算,即可实现零样本的仿真到现实迁移并具备鲁棒的真实世界表现。我们的模拟器FiGS结合了计算简单的无人机动力学模型与高视觉保真的高斯泼溅场景重建,能以最高130帧/秒的速度快速生成逼真图像。我们利用FiGS,通过具有特权状态与动力学信息的专家模型,收集了10万至30万张图像/状态-动作对,涵盖动力学参数与空间扰动的随机化。随后,我们将该专家策略蒸馏为一个轻量级神经网络——SV-Net。SV-Net在20赫兹频率下处理彩色图像、光流和IMU数据流,生成低层推力与机体速率指令。关键在于,SV-Net包含一个可实时自适应动态变化的低层控制学习模块。在105次硬件实验中,我们验证了SOUS VIDE策略对30%质量变化、40米/秒风速突变、60%环境亮度变化、场景中物体移动或移除以及人员剧烈穿行均表现出鲁棒性。代码、数据及实验视频见项目主页:https://stanfordmsl.github.io/SousVide/
原文摘要 · Abstract (English)
We propose a new simulator, training approach, and policy architecture, collectively called SOUS VIDE, for end-to-end visual drone navigation. Our trained policies exhibit zero-shot sim-to-real transfer with robust real-world performance using only onboard perception and computation. Our simulator, called FiGS, couples a computationally simple drone dynamics model with a high visual fidelity Gaussian Splatting scene reconstruction. FiGS can quickly simulate drone flights producing photorealistic images at up to 130 fps. We use FiGS to collect 100k-300k image/state-action pairs from an expert MPC with privileged state and dynamics information, randomized over dynamics parameters and spatial disturbances. We then distill this expert MPC into an end-to-end visuomotor policy with a lightweight neural architecture, called SV-Net. SV-Net processes color image, optical flow and IMU data streams into low-level thrust and body rate commands at 20 Hz onboard a drone. Crucially, SV-Net includes a learned module for low-level control that adapts at runtime to variations in drone dynamics. In a campaign of 105 hardware experiments, we show SOUS VIDE policies to be robust to 30% mass variations, 40 m/s wind gusts, 60% changes in ambient brightness, shifting or removing objects from the scene, and people moving aggressively through the drone's visual field. Code, data, and experiment videos can be found on our project page: https://stanfordmsl.github.io/SousVide/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。