用轻量级方法让通用视觉模型理解多光谱和雷达图像。
Lightweight Adaptation of General-Purpose VLMs for Multispectral and SAR Image Understanding

- 将多光谱和雷达数据转为多个虚拟视角输入,通过提示词引导模型理解。
- 在六类地物分类任务上达到0.8275的微平均F1,优于传统方法。
- 仅用LoRA微调即可适配多种模型,适合遥感与跨模态研究者。
通用视觉语言模型(VLMs)具备强大的视觉识别、指令遵循和生成能力,但其预训练视觉编码器通常基于三通道自然图像,难以直接处理多光谱或合成孔径雷达(SAR)等观测数据。现有适配方法需专用编码器和领域预训练,限制了通用模型的复用。本文提出一种轻量级方案:利用通用VLM的多图像接口,将每项观测渲染为五个光学视图和一个SAR视图,通过提示词命名并结合LoRA对语言网络及部分视觉变压器块进行微调。该方法可有效传递波段组合、光谱指数和雷达后向散射信息。针对地物分类任务,采用结构化监督,将预测类别与传感器证据关联;进一步构建偏好对,隐去真实标签但保留支持证据,促使模型生成完整且一致的预测。在基于BigEarthNet-v2的六类地物分类基准上,适配后的Qwen3-VL达到0.8275的微平均F1。相同输入与适配协议提升所有四款测试的VLM架构性能,并可迁移至Sen1Floods11洪水验证与BigEarthNet.txt图像描述任务。图像移除与不匹配控制实验表明,模型确实依赖提供的传感器观测。结果证明,通过渲染输入与紧凑的LoRA适配,无需训练新基础模型即可重用VLM完成多光谱与SAR任务。
原文摘要 · Abstract (English)
General-purpose vision-language models (VLMs) now support strong visual recognition, instruction following, and generation. However, most pretrained visual encoders are built around three-channel natural images and do not directly accommodate observations such as native multispectral measurements or synthetic aperture radar (SAR). Adapting VLMs to these sensors typically requires dedicated encoders and domain pretraining, slowing the reuse of stronger general-purpose checkpoints. We show that the multi-image interface of general-purpose VLMs offers a lightweight alternative. Our protocol renders each observation as five optical views and one SAR view, names them in the prompt, and adapts the language network and selected visual transformer blocks with LoRA. This exposes band composites, spectral indices, and radar backscatter through an existing visual interface. For land-cover recognition, structured supervision couples predicted classes with sensor evidence. We further construct preference pairs in which a true label is omitted while its supporting evidence is retained, encouraging complete predictions that remain consistent with the observations. On a balanced six-class land-cover benchmark derived from BigEarthNet-v2, the adapted Qwen3-VL reaches 0.8275 micro F1. The same input and adaptation protocol improves all four tested VLM architectures and transfers to Sen1Floods11 flood verification and BigEarthNet.txt captioning. Image removal and mismatch controls show that the adapted models use the supplied sensor observations. Together, these results demonstrate that VLMs can be repurposed for multispectral and SAR tasks through rendered inputs and compact LoRA adaptation, without training a new foundation model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。