针对遥感模型模态不匹配问题,提出可按需剪枝的多模态专家模型。
MAPEX: Modality-Aware Pruning of Experts for Remote Sensing Foundation Models
- 用模态感知路由机制激活特定模态专家,实现多模态融合
- 剪枝后模型在多个遥感数据集上性能优于全监督训练和现有基线
- 适合小样本任务,显著降低微调与部署成本
遥感数据广泛应用于洪水测绘、野火检测和土地利用研究。传统方法针对不同任务选择特定模态或专用仪器数据。近期遥感基础模型通过大规模遥感数据预训练计算机视觉模型,但多数聚焦于光学RGB或高光谱数据,导致应用模态与预训练数据不匹配。此外,基础模型规模庞大,难以在通常较小的任务数据集上微调。本文提出MAPEX,一种基于多模态专家混合的基础模型,其采用新颖的模态条件令牌路由机制,激发针对特定模态的专家。针对具体任务时,提出模态感知剪枝技术,仅保留与任务模态相关的专家。该方法生成高效且专用于特定模态的模型,简化了微调与部署流程。我们在多个遥感数据集上验证了MAPEX的有效性,结果表明其性能优于全监督训练及当前最优遥感基础模型。代码已开源:https://github.com/HSG-AIML/MAPEX。
原文摘要 · Abstract (English)
Remote sensing data is commonly used for tasks such as flood mapping, wildfire detection, or land-use studies. For each task, scientists carefully choose appropriate modalities or leverage data from purpose-built instruments. Recent work on remote sensing foundation models pre-trains computer vision models on large amounts of remote sensing data. These large-scale models tend to focus on specific modalities, often optical RGB or multispectral data. For many important applications, this introduces a mismatch between the application modalities and the pre-training data. Moreover, the large size of foundation models makes them expensive and difficult to fine-tune on typically small datasets for each task. We address this mismatch with MAPEX, a remote sensing foundation model based on mixture-of-modality experts. MAPEX is pre-trained on multi-modal remote sensing data with a novel modality-conditioned token routing mechanism that elicits modality-specific experts. To apply the model on a specific task, we propose a modality aware pruning technique, which only retains experts specialized for the task modalities. This yields efficient modality-specific models while simplifying fine-tuning and deployment for the modalities of interest. We experimentally validate MAPEX on diverse remote sensing datasets and show strong performance compared to fully supervised training and state-of-the-art remote sensing foundation models. Code is available at https://github.com/HSG-AIML/MAPEX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。