用3D高斯场+对比学习,让无人机视觉控制模型更省力、更抗干扰。
GaussFly: Contrastive Reinforcement Learning for Visuomotor Policies in 3D Gaussian Fields
- 先用3D高斯点云重建真实场景,再在逼真仿真中训练
- 通过对比学习提取抗噪特征,使策略更高效且性能更强
- 零样本迁移至复杂真实环境,显著缩小仿真到现实的差距
仅依赖单目视觉的自主飞行器视觉-运动策略学习既吸引人又极具挑战性。现有端到端方法直接将高维RGB观测映射为动作指令,常因样本效率低和仿真到现实的视觉差异而表现不佳。为此,我们提出GaussFly框架,通过‘真实→仿真→真实’的协同范式,显式解耦表征学习与策略优化。首先,利用加入几何约束的3D高斯泼溅(3DGS)重建训练场景,实现高保真真实到仿真转换;其次,基于这些逼真的仿真环境,采用对比表示学习从渲染的RGB图像中提取紧凑、抗噪的潜在特征。使用预训练编码器提供低维特征输入,显著降低视觉运动策略的计算负担,同时提升其对视觉噪声的鲁棒性。大量仿真与真实世界实验表明,GaussFly相比基线方法具有更优的样本效率和最终性能,关键在于能实现对未见真实环境的鲁棒零样本策略迁移,有效弥合仿真到现实的鸿沟。
原文摘要 · Abstract (English)
Learning visuomotor policies for Autonomous Aerial Vehicles (AAVs) relying solely on monocular vision is an attractive yet highly challenging paradigm. Existing end-to-end learning approaches directly map high-dimensional RGB observations to action commands, which frequently suffer from low sample efficiency and severe sim-to-real gaps due to the visual discrepancy between simulation and physical domains. To address these long-standing challenges, we propose GaussFly, a novel framework that explicitly decouples representation learning from policy optimization through a cohesive real-to-sim-to-real paradigm. First, to achieve a high-fidelity real-to-sim transition, we reconstruct training scenes using 3D Gaussian Splatting (3DGS) augmented with explicit geometric constraints. Second, to ensure robust sim-to-real transfer, we leverage these photorealistic simulated environments and employ contrastive representation learning to extract compact, noise-resilient latent features from the rendered RGB images. By utilizing this pre-trained encoder to provide low-dimensional feature inputs, the computational burden on the visuomotor policy is significantly reduced while its resistance against visual noise is inherently enhanced. Extensive experiments in simulated and real-world environments demonstrate that GaussFly achieves superior sample efficiency and asymptotic performance compared to baselines. Crucially, it enables robust and zero-shot policy transfer to unseen real-world environments with complex textures, effectively bridging the sim-to-real gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。