用检索增强学习缩小真实与仿真自动驾驶数据差距,提升模型泛化能力。
RALAD: Bridging the Real-to-Sim Domain Gap in Autonomous Driving with Retrieval-Augmented Learning
- 基于优化的最优传输方法,同时考虑图像个体与组间距离进行域适应。
- 在仿真环境中使mIOU提升10.30%、mAP提升12.29%,真实场景性能稳定。
- 冻结高计算成本层,训练成本降低约88.1%,适配多种模型架构。
为构建鲁棒的自动驾驶系统,真实世界训练的模型在新环境(如极端天气)下常表现不佳。由于真实世界收集罕见场景困难,需依赖仿真验证,但高计算开销与数据分布差异导致真实与仿真场景难以无缝迁移。为此,我们提出面向自动驾驶的检索增强学习框架RALAD,以低成本弥合真实-仿真域差距。RALAD包含三项核心设计:(1) 通过改进的最优传输(OT)方法实现域适应,兼顾单张图像与图像组间的距离;(2) 简洁统一的框架,可适配多种模型;(3) 高效微调策略,冻结高耗计算层的同时保持模型鲁棒性。实验表明,以Cross View为例,经RALAD微调后,真实场景中mIOU和mAP保持稳定,仿真环境中分别提升10.30%和12.29%。整体重训练成本降低约88.1%。代码已开源:https://github.com/JiachengZuo/RALAD.git。
原文摘要 · Abstract (English)
In the pursuit of robust autonomous driving systems, models trained on real-world datasets often struggle to adapt to new environments, particularly when confronted with corner cases such as extreme weather conditions. Collecting these corner cases in the real world is non-trivial, which necessitates the use of simulators for validation. However,the high computational cost and the domain gap in data distribution have hindered the seamless transition between real and simulated driving scenarios. To tackle this challenge, we propose Retrieval-Augmented Learning for Autonomous Driving (RALAD), a novel framework designed to bridge the real-to-sim gap at a low cost. RALAD features three primary designs, including (1) domain adaptation via an enhanced Optimal Transport (OT) method that accounts for both individual and grouped image distances, (2) a simple and unified framework that can be applied to various models, and (3) efficient fine-tuning techniques that freeze the computationally expensive layers while maintaining robustness. Experimental results demonstrate that RALAD compensates for the performance degradation in simulated environments while maintaining accuracy in real-world scenarios across three different models. Taking Cross View as an example, the mIOU and mAP metrics in real-world scenarios remain stable before and after RALAD fine-tuning, while in simulated environments,the mIOU and mAP metrics are improved by 10.30% and 12.29%, respectively. Moreover, the re-training cost of our approach is reduced by approximately 88.1%. Our code is available at https://github.com/JiachengZuo/RALAD.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。