将视觉大模型迁移至非自然图像模态，显著提升分割性能
SimCMF: A Simple Cross-modal Fine-tuning Strategy from Vision Foundation Models to Any Imaging Modality
- 设计跨模态对齐模块，解决不同成像物理特性间的偏差问题
- 在多种新型成像模态上平均提升分割准确率至53.88%（原22.15%）
- 适用于无基准场景，为多模态传感器提供通用迁移方案
如ChatGPT和Sora等大规模训练的基座模型已带来革命性影响，但众多领域传感器难以收集足够规模的自然图像来训练强基座模型。为此，本文提出SimCMF框架，研究从自然RGB图像训练的视觉基座模型向其他物理特性不同的成像模态（如偏振）进行跨模态微调的问题。通过系统分析基础组件，提出新型跨模态对齐模块以缓解模态错配。将SimCMF应用于代表性基座模型SAM，支持任意新成像模态评估。因缺乏相关基准，构建了专用评测集。实验表明，该方法可使所评估模态的分割性能（mIoU）从22.15%平均提升至53.88%，持续优于现有基线。代码已开源。
原文摘要 · Abstract (English)
Foundation models like ChatGPT and Sora that are trained on a huge scale of data have made a revolutionary social impact. However, it is extremely challenging for sensors in many different fields to collect similar scales of natural images to train strong foundation models. To this end, this work presents a simple and effective framework, SimCMF, to study an important problem: cross-modal fine-tuning from vision foundation models trained on natural RGB images to other imaging modalities of different physical properties (e.g., polarization). In SimCMF, we conduct a thorough analysis of different basic components from the most naive design and ultimately propose a novel cross-modal alignment module to address the modality misalignment problem. We apply SimCMF to a representative vision foundation model Segment Anything Model (SAM) to support any evaluated new imaging modality. Given the absence of relevant benchmarks, we construct a benchmark for performance evaluation. Our experiments confirm the intriguing potential of transferring vision foundation models in enhancing other sensors' performance. SimCMF can improve the segmentation performance (mIoU) from 22.15% to 53.88% on average for evaluated modalities and consistently outperforms other baselines. The code is available at https://github.com/mt-cly/SimCMF
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。