arXiv:2511.16998cs.CV2025-11

用视觉语言模型增强图像修复,提升恶劣天气下图像质量。

VLM-Augmented Degradation Modeling for Image Restoration Under Adverse Weather Conditions

  • 结合视觉语言模型与隐式记忆库,动态检索降质原型。
  • 在四个极端天气数据集上PSNR和SSIM均优于基线方法。
  • 模型轻量高效,适合自动驾驶等实时场景部署。

在雨、雾、雪或其混合等恶劣天气条件下实现可靠的视觉感知对自动驾驶和户外机器人至关重要但极具挑战。本文提出统一的内存增强型视觉-语言恢复(MVLR)模型,可恢复不同降质程度下的多种天气条件图像。MVLR将轻量编码器-解码器主干与视觉语言模型(VLM)及隐式记忆库(IMB)结合:VLM通过思维链推理编码天气降质先验,IMB存储降质模式的连续潜在表示;由VLM生成的先验查询IMB以检索细粒度降质原型,并通过动态交叉注意力机制自适应融合至多尺度视觉特征,从而提升修复精度并保持计算效率。在四个严重天气基准上的大量实验表明,MVLR在峰值信噪比(PSNR)和结构相似性指数(SSIM)方面均超越单分支与专家混合基线模型。结果表明,MVLR在模型紧凑性与表达能力之间实现了实用平衡,适用于多样户外环境中的实时部署。

原文摘要 · Abstract (English)

Reliable visual perception under adverse weather conditions, such as rain, haze, snow, or a mixture of them, is desirable yet challenging for autonomous driving and outdoor robots. In this paper, we propose a unified Memory-Enhanced Visual-Language Recovery (MVLR) model that restores images from different degradation levels under various weather conditions. MVLR couples a lightweight encoder-decoder backbone with a Visual-Language Model (VLM) and an Implicit Memory Bank (IMB). The VLM performs chain-of-thought inference to encode weather degradation priors and the IMB stores continuous latent representations of degradation patterns. The VLM-generated priors query the IMB to retrieve fine-grained degradation prototypes. These prototypes are then adaptively fused with multi-scale visual features via dynamic cross-attention mechanisms, enhancing restoration accuracy while maintaining computational efficiency. Extensive experiments on four severe-weather benchmarks show that MVLR surpasses single-branch and Mixture-of-Experts baselines in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM). These results indicate that MVLR offers a practical balance between model compactness and expressiveness for real-time deployment in diverse outdoor conditions.

图像修复视觉语言模型恶劣天气轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。