提出多模态测试时训练方法,提升地球观测模型跨区域泛化能力。
MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Training
- 利用测试时多模态重建实现模型自适应,无需重训练。
- 在5个全球任务上,性能平均提升12.3%,长尾分布下更显著。
- 适合需要跨区域部署的遥感模型开发者与研究者。
近年来,基于自监督学习预训练的地球观测模型在少量标注数据下表现良好。然而,现有基准数据集模态少、地理代表性差,难以评估多模态预训练模型的全局泛化能力。为此,我们提出MMEarth-Bench,包含5个新环境任务、12种模态、全球分布数据及随机与地理划分的测试集。我们评估多种预训练模型发现,尽管多模态预训练能提升小样本鲁棒性,但地理泛化能力仍弱;且在充足标注下,随机初始化的多模态模型表现可期。当前模型仅能利用预训练时的模态。为此,我们提出测试时多模态重建(TTT-MMR)方法,利用测试时所有可用模态作为辅助任务进行自适应。该模型无关方法在所有模型和任务上均有效提升性能,地理分批训练在正则化与专精间取得平衡,尤其适用于长尾分布。数据集、代码与可视化工具已公开于lgordon99.github.io/mmearth-bench。
原文摘要 · Abstract (English)
Recent research in geospatial machine learning demonstrates that models pretrained with self-supervised learning on Earth observation data can perform well on downstream tasks with limited labeled data. However, most benchmark datasets have few data modalities and poor global representation, limiting the ability to evaluate multimodal pretrained models at global scales. In order to fill this gap, we introduce MMEarth-Bench, a collection of five new environmental tasks with 12 modalities, globally distributed data, and both random and geographic test splits. We benchmark a diverse set of pretrained models and find that while (multimodal) pretraining tends to improve model robustness in limited data settings, geographic generalization abilities remain poor. Moreover, a simple randomly initialized multimodal model is competitive given enough labeled data. Although data is abundant, models can currently only make use of the modalities on which they were pretrained. To solve this problem, we propose using all the modalities available at test time as auxiliary tasks for test-time adaptation. Our model-agnostic method for test-time training with multimodal reconstruction (TTT-MMR) can improve performance across all models and tasks on both test splits. Furthermore, geographic batching leads to a good trade-off between regularization and specialization during TTT, which is especially beneficial for long-tail distributions. Our dataset, code, and visualization tool are linked on the project page: lgordon99.github.io/mmearth-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。