用有效秩指导微调,让模型在恶劣天气下也能准确估深度。
ER-LoRA: Effective-Rank Guided Adaptation for Weather-Generalized Depth Estimation
- 基于有效秩分解视觉基础模型权重,分阶段选择性微调。
- 仅用少量正常天气数据,精度超越全量微调和合成数据训练方法。
- 适合追求高效适配且保持模型泛化能力的研究者或开发者。
单目深度估计在雨、雾、雪和夜间等恶劣天气下仍极具挑战,主要源于缺乏可靠真实标签,且难以从无标注的真实世界数据中学习。现有方法多依赖带伪标签的合成恶劣数据,存在领域差异;或采用自监督学习,但在恶劣场景下违反光照一致性假设。本文提出通过参数高效微调(PEFT)视觉基础模型(VFM),仅使用少量高可见度(正常)数据实现天气泛化的深度估计。针对几何任务中PEFT研究不足的问题,我们设计了选择-微调-保持(STM)策略,基于熵秩与稳定秩两种有效秩结构分解预训练权重。微调阶段根据熵秩和全微调权重自适应选择秩数与任务感知奇异方向进行初始化;保持阶段则基于稳定秩施加主方向正则化,确保灵活适应同时保留预训练知识的强泛化能力。在四个跨多样天气的真实世界基准上大量实验表明,STM不仅优于现有PEFT方法与全微调,还超越了使用合成恶劣数据的方法,甚至超过深度基础模型。
原文摘要 · Abstract (English)
Monocular depth estimation under adverse weather conditions (e.g.\ rain, fog, snow, and nighttime) remains highly challenging due to the lack of reliable ground truth and the difficulty of learning from unlabeled real-world data. Existing methods often rely on synthetic adverse data with pseudo-labels, which suffer from domain gaps, or employ self-supervised learning, which violates photometric assumptions in adverse scenarios. In this work, we propose to achieve weather-generalized depth estimation by Parameter-Efficient Fine-Tuning (PEFT) of Vision Foundation Models (VFMs), using only a small amount of high-visibility (normal) data. While PEFT has shown strong performance in semantic tasks such as segmentation, it remains underexplored for geometry -- centric tasks like depth estimation -- especially in terms of balancing effective adaptation with the preservation of pretrained knowledge. To this end, we introduce the Selecting-Tuning-Maintaining (STM) strategy, which structurally decomposes the pretrained weights of VFMs based on two kinds of effective ranks (entropy-rank and stable-rank). In the tuning phase, we adaptively select the proper rank number as well as the task-aware singular directions for initialization, based on the entropy-rank and full-tuned weight; while in the maintaining stage, we enforce a principal direction regularization based on the stable-rank. This design guarantees flexible task adaptation while preserving the strong generalization capability of the pretrained VFM. Extensive experiments on four real-world benchmarks across diverse weather conditions demonstrate that STM not only outperforms existing PEFT methods and full fine-tuning but also surpasses methods trained with adverse synthetic data, and even the depth foundation model
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。