让遥感大模型在有云雾和乱句时仍能稳定推理。
RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observation

- 用干净与带噪数据配对训练,通过对比学习增强鲁棒性。
- 在三类遥感任务中,对云雾和模糊指令的抗干扰能力显著提升。
- 适合实际部署中面对复杂输入的遥感智能系统开发者。
面向地球观测的鲁棒多模态大语言模型需在真实输入变化下保持一致的理解与推理能力。然而现有遥感多模态大模型因在精心清洗的数据集上训练,学习到的是脆弱的映射关系,难以泛化至实际操作中的噪声环境。为此,我们构建了一套包含云雾遮挡等视觉退化以及口语化、模糊或缺失指令等文本变异的多模态扰动集。实证评估表明,这些扰动严重削弱主流遥感基础模型的视觉-语义推理能力。为此,我们提出RemoteShield,一种在真实输入变化下保持输出一致性的遥感多模态大模型。训练时,每个干净样本与其对应的图像-文本扰动变体组成语义等价簇,模型通过偏好学习在同簇内比较清洁与扰动输入的响应,鼓励选择稳定输出而非受扰动影响的结果。这种跨条件对齐使模型聚焦于任务本质语义,即使在视觉退化与文本噪声下仍保持一致。三个地球观测任务实验显示,RemoteShield在真实多模态扰动下均显著优于代表性基线,表现出更强的鲁棒性与跨条件一致性。
原文摘要 · Abstract (English)
A robust Multimodal Large Language Model (MLLM) for Earth Observation should maintain consistent interpretation and reasoning under realistic input variations. However, current Remote Sensing MLLMs fail to meet this requirement. Trained on carefully curated clean datasets, they learn brittle mappings that do not generalize to noisy conditions in operational Earth Observation. Consequently, their performance degrades when confronted with imperfect inputs in deployment. To quantify this vulnerability, we construct a realistic set of multimodal perturbations, including visual degradations such as cloud and fog cover, together with diverse human-centric textual variations ranging from colloquialisms to vague or omitted instructions. Empirical evaluations show that these perturbations significantly impair the visual-semantic reasoning capabilities of leading RS foundation models. To address this limitation, we introduce RemoteShield, a robust Remote Sensing MLLM trained to maintain consistent outputs across realistic input variations. During training, each clean sample is paired with its image-text perturbed variants to form a semantic equivalence cluster. Rather than directly fitting noisy samples, RemoteShield is optimized through preference learning over clean and perturbed conditions within the same cluster. By comparing model responses to clean and corrupted inputs, the model is encouraged to favor stable responses over perturbation-induced failures. This cross-condition alignment helps the model focus on underlying task semantics despite visual degradations and textual noise. Experiments on three Earth Observation tasks show that RemoteShield consistently delivers stronger robustness and cross-condition consistency than representative baselines under realistic multimodal perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。