让预训练视觉模型直接处理相机RAW图像,提升真实场景适应能力。
RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images and A Benchmark
- 用可学习的ISP模块作为输入适配器,融合到预训练模型中。
- 在17类真实退化场景下性能超越现有方法,尤其在光照和天气变化中更鲁棒。
- 适合需要高保真图像理解的领域,如自动驾驶、医学成像。
当前计算机视觉主流偏好使用sRGB图像进行预训练,因其获取便捷且存储紧凑。然而,相机RAW图像在多样真实场景中保留了丰富的物理细节。尽管如此,多数现有感知方法将图像信号处理(ISP)与后续网络模块耦合,忽视了模型层面的协同优化。本文提出RAW-Adapter,一种基于适配器的方法:在输入端引入可学习的ISP模块作为适配器,同时在模型层使用适配器实现ISP与高层架构的无缝衔接。该框架可适配多种视觉任务。此外,我们构建了包含17类基于RAW的常见退化(如亮度失真、天气影响、模糊、相机成像缺陷及色彩响应差异)的基准测试集RAW-Bench。通过该基准,系统比较了RAW-Adapter与先进ISP方法及其他基于RAW的高层视觉算法的表现。我们还提出一种基于RAW的数据增强策略,进一步提升其在域外(OOD)场景下的泛化能力。大量实验验证了RAW-Adapter的有效性与高效性,展现了其在多样化场景中的强鲁棒性。
原文摘要 · Abstract (English)
In the computer vision community, the preference for pre-training visual models has largely shifted toward sRGB images due to their ease of acquisition and compact storage. However, camera RAW images preserve abundant physical details across diverse real-world scenarios. Despite this, most existing visual perception methods that utilize RAW data directly integrate image signal processing (ISP) stages with subsequent network modules, often overlooking potential synergies at the model level. Building on recent advances in adapter-based methodologies in both NLP and computer vision, we propose RAW-Adapter, a novel framework that incorporates learnable ISP modules as input-level adapters to adjust RAW inputs. At the same time, it employs model-level adapters to seamlessly bridge ISP processing with high-level downstream architectures. Moreover, RAW-Adapter serves as a general framework applicable to various computer vision frameworks. Furthermore, we introduce RAW-Bench, which incorporates 17 types of RAW-based common corruptions, including lightness degradations, weather effects, blurriness, camera imaging degradations, and variations in camera color response. Using this benchmark, we systematically compare the performance of RAW-Adapter with state-of-the-art (SOTA) ISP methods and other RAW-based high-level vision algorithms. Additionally, we propose a RAW-based data augmentation strategy to further enhance RAW-Adapter's performance and improve its out-of-domain (OOD) generalization ability. Extensive experiments substantiate the effectiveness and efficiency of RAW-Adapter, highlighting its robust performance across diverse scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。