让遥感模型自动适应新旧传感器，无需重新训练
Changing Modalities: Adapting Remote Sensing Models to New Satellites and Sensors
- 用模块化结构实现模态替换、增加和缩减的统一处理
- 通过幻觉生成缺失模态，在无标签数据上端到端训练
- 适合卫星换代时快速部署模型，节省标注与计算成本
遥感领域的机器学习模型通常在固定的模态组合上训练和部署。但随着新卫星配备新型传感器、旧传感器退役，从业者可能希望在有限数据或计算资源下，将现有模型迁移到新的模态组合（如替代、扩展或子集）。本文研究了模态变化下的模型更新问题，识别出三种主要场景：模态迁移（替换）、新增（扩展）和窥探（子集）。为此提出 DeluluNet，一种支持所有三种场景的模块化架构。该模型通过模态幻觉机制，利用未标记的多模态数据，从单模态教师模型中学习多模态表示——即根据已有的模态预测缺失模态的特征。因此，即使输入模态发生变化，模型仍能持续输出，为不断变化的遥感环境提供无需重新标注和训练的实用解决方案。
原文摘要 · Abstract (English)
Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination--predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。