arXiv:2501.12535cs.LGcs.CV2025-01中稿 · AAAI被引 7

数据分布影响地学模型性能,全球均衡采样更优。

How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?

  • 从全球数据池中采样不同地理分布数据测试模型表现
  • 均衡覆盖的训练数据使模型在下游任务中表现更佳
  • 研究结果可指导地学大模型的数据构建策略

基础模型在地球观测领域快速发展,地学基础模型(GFMs)有助于应对气候变化、农业和灾害响应等全球挑战。以往研究聚焦于模型架构与预训练任务设计,未关注预训练数据选择对性能的影响。然而,其他领域的研究表明,预训练数据分布是影响基础模型性能的关键因素。为此,本文探索了预训练数据的地理分布如何影响GFMs性能。通过从全球数据池中采样不同组合,评估了多种数据分布策略。在两个GFM上进行的下游任务实验表明,平衡且具有全球代表性的数据组合通常优于区域特异性采样,凸显了多样性与全球覆盖的重要性。结果还显示,最合适的采样方法可能依赖于具体模型架构。这些发现将助力构建鲁棒的GFMs,通过优化预训练数据分布提升机器学习在地球观测中的应用效果。

原文摘要 · Abstract (English)

Foundation models have made rapid advances in many domains including Earth observation, where Geospatial Foundation Models (GFMs) can help address global challenges such as climate change, agriculture, and disaster response. Previous work on GFMs focused on tailoring model architecture and pre-text tasks, and did not investigate the impact of pre-training data selection on model performance. However, recent works from other domains show that the pre-training data distribution is an important factor influencing the performance of the foundation models. With this motivation, our research explores how the geographic distribution of pre-training data affects the performance of GFMs. We evaluated several pre-training data distributions by sampling different compositions from a global data pool. Our experiments with two GFMs on downstream tasks indicate that balanced and globally representative data compositions often outperform region-specific sampling, highlighting the importance of diversity and global coverage in pre-training data. Our results suggest that the most appropriate data sampling technique may depend on the specific GFM architecture. These findings will support the development of robust GFMs by incorporating quality pre-training data distributions, ultimately improving machine learning solutions for Earth observation.

地学模型数据分布基础模型地球观测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。