arXiv:2412.20162cs.CV2024-12

用多模态驱动的低秩适配,提升恶劣天气下深度估计精度。

Multi-Modality Driven LoRA for Adverse Condition Depth Estimation

  • 通过低秩矩阵实现高效微调,减少模型复杂度。
  • 在nuScenes和Oxford RobotCar上达到当前最优效果。
  • 适合自动驾驶中恶劣天气场景的深度感知研究者。

自动驾驶领域日益关注边缘场景问题,尤其是恶劣天气(如夜间、雾天、雨天)下的行车安全。为此,恶劣条件深度估计(ACDE)任务受到广泛关注。以往方法主要依赖生成模型,需额外目标图像将晴天转换为恶劣天气,或使用可学习参数进行特征增强以弥合域间差距,导致模型复杂度和调参成本增加。此外,与CLIP中图文特征预先对齐不同,深度估计模型缺乏多模态特征间的充分对齐,影响了在恶劣条件下的语义理解一致性。为此,本文提出多模态驱动的低秩适配(MMD-LoRA),利用低秩适配矩阵实现从源域到目标域的高效微调。其包含两个核心组件:提示驱动域对齐(PDDA)与视觉-文本一致对比学习(VTCCL)。PDDA通过语言与图像间的源-目标差异相等性约束,引导图像编码器生成目标域视觉表示;VTCCL则桥接CLIP的文本特征与扩散模型的视觉特征,拉远不同天气的表示,拉近相似天气的表示。大量实验表明,该方法在nuScenes和Oxford RobotCar数据集上均达到当前最优性能,展现出对多样化恶劣环境的强大适应能力与高效性。

原文摘要 · Abstract (English)

The autonomous driving community is increasingly focused on addressing corner case problems, particularly those related to ensuring driving safety under adverse conditions (e.g., nighttime, fog, rain). To this end, the task of Adverse Condition Depth Estimation (ACDE) has gained significant attention. Previous approaches in ACDE have primarily relied on generative models, which necessitate additional target images to convert the sunny condition into adverse weather, or learnable parameters for feature augmentation to adapt domain gaps, resulting in increased model complexity and tuning efforts. Furthermore, unlike CLIP-based methods where textual and visual features have been pre-aligned, depth estimation models lack sufficient alignment between multimodal features, hindering coherent understanding under adverse conditions. To address these limitations, we propose Multi-Modality Driven LoRA (MMD-LoRA), which leverages low-rank adaptation matrices for efficient fine-tuning from source-domain to target-domain. It consists of two core components: Prompt Driven Domain Alignment (PDDA) and Visual-Text Consistent Contrastive Learning(VTCCL). During PDDA, the image encoder with MMD-LoRA generates target-domain visual representations, supervised by alignment loss that the source-target difference between language and image should be equal. Meanwhile, VTCCL bridges the gap between textual features from CLIP and visual features from diffusion model, pushing apart different weather representations (vision and text) and bringing together similar ones. Through extensive experiments, the proposed method achieves state-of-the-art performance on the nuScenes and Oxford RobotCar datasets, underscoring robustness and efficiency in adapting to varied adverse environments.

深度估计多模态低秩适配自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。