首个评估遥感模型真实分布偏移鲁棒性的公开基准，揭示当前模型在实际应用中表现显著下降。
EarthShift: a benchmark for measuring robustness to real-world distribution shifts in Earth observation
- 构建跨时间、地理、传感器等多维度的遥感数据集对，对比模型分布内/外表现差异。
- 8个地学基础模型在11个任务上平均出域性能下降15%-20%，与通用视觉模型相当。
- 适合关注遥感模型真实部署可靠性、需提升泛化能力的研究者使用。
当前遥感基准主要衡量模型在多样化任务中的性能，通常仅关注分布内泛化。然而模型部署时需应对多种分布外场景，如新时间段、地理区域、尺度和传感器。我们提出EarthShift：首个用于评估遥感中真实分布偏移鲁棒性的公开测试平台。EarthShift通过来自不同来源、时间窗口、地理区域和传感器的配对数据集，使用户能够比较模型在分布内与分布外的表现。我们在8个地学基础模型（GFMs）和11个任务上，覆盖5种偏移类型进行实验，结果表明，无论模型架构、规模、预训练或微调策略如何，GFMs平均出域性能下降15%-20%。研究显示，地学基础模型的鲁棒性与通用视觉基础模型甚至全监督模型相当。这凸显了未来研究应更重视分布鲁棒性提升，而非仅追求性能。EarthShift代码与数据已开源，供后续研究使用。
原文摘要 · Abstract (English)
Current Earth observation benchmarks focus on measuring performance on diverse tasks and applications, typically measuring generalization in-distribution. But when models are deployed, they must generalize to myriad out-of-distribution scenarios, such as new time periods, geographies, scales, and sensors. We introduce EarthShift: the first public testbed for benchmarking robustness across multiple realistic distribution shifts encountered in remote sensing. EarthShift enables users to measure distributional robustness by comparing performance in- and out-of-distribution using datasets from paired datasets from different sources, temporal windows, geographic locations, and sensors. Our experiments on 8 geospatial foundation models (GFMs) and 11 tasks covering 5 shift types show that GFMs consistently perform 15-20% worse out-of-distribution on average regardless of model architecture, size, pre-training or fine-tuning strategy. We show that GFM robustness is similar to that of generic vision foundation models, and even fully-supervised models. This highlights a need for future research to strive for improvements in distributional robustness, not just performance, which can be benchmarked using EarthShift. We release our code and datasets to provide a testbed to guide future work to create foundation models that are robust and reliable in real-world applications. Code and data for EarthShift are available at: https://earthshift.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。