用低秩适配让预训练模型快速应对新天气,保持原有性能。
Low-rank Adaptation-based All-Weather Removal for Autonomous Navigation
- 用低秩适配(LoRA)高效微调预训练模型,无需重新训练。
- 提出LoRA-A方法,通过奇异值分解对齐权重向量,减少性能退化。
- 在自动驾驶任务中验证,恢复图像可有效支持语义分割与深度估计。
恶劣天气下图像恢复(AWIR)对自动驾驶可靠性至关重要。现有模型通常针对特定天气(如雾、雨、雪)训练,难以应对分布外(OoD)样本或未见退化,导致实际应用受限。传统方法需重训练或微调,前者依赖大规模数据,后者参数量大且不高效。本文提出使用低秩适配(LoRA)高效适应预训练模型至新天气任务。进一步发现,常规LoRA会降低模型在原任务上的表现。为此,我们提出基于奇异值分解(SVD)的LoRA-Align(LoRA-A)方法,通过对齐微调前后权重矩阵的奇异向量,保留原任务知识同时适应新任务。实验表明,经LoRA与LoRA-A恢复的图像能有效用于自动驾驶中的语义分割与深度估计。
原文摘要 · Abstract (English)
All-weather image restoration (AWIR) is crucial for reliable autonomous navigation under adverse weather conditions. AWIR models are trained to address a specific set of weather conditions such as fog, rain, and snow. But this causes them to often struggle with out-of-distribution (OoD) samples or unseen degradations which limits their effectiveness for real-world autonomous navigation. To overcome this issue, existing models must either be retrained or fine-tuned, both of which are inefficient and impractical, with retraining needing access to large datasets, and fine-tuning involving many parameters. In this paper, we propose using Low-Rank Adaptation (LoRA) to efficiently adapt a pre-trained all-weather model to novel weather restoration tasks. Furthermore, we observe that LoRA lowers the performance of the adapted model on the pre-trained restoration tasks. To address this issue, we introduce a LoRA-based fine-tuning method called LoRA-Align (LoRA-A) which seeks to align the singular vectors of the fine-tuned and pre-trained weight matrices using Singular Value Decomposition (SVD). This alignment helps preserve the model's knowledge of its original tasks while adapting it to unseen tasks. We show that images restored with LoRA and LoRA-A can be effectively used for computer vision tasks in autonomous navigation, such as semantic segmentation and depth estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。