arXiv:2410.07472cs.LGcs.AI2024-10被引 12

系统分析深度学习天气预报设计选择,发现固定网格模型更优,多步微调关键。

Exploring the design space of deep-learning-based weather forecasting systems

  • 对比多种架构与训练策略,评估其对天气预报性能影响。
  • 固定网格模型显著优于网格无关模型,多步微调对性能至关重要。
  • 小模型用大数据更有效,大模型需更大算力,适合气象建模研究者。

尽管基于深度学习的天气预报系统取得了巨大进展,但其设计空间——包括不同设计选择的影响——仍不清晰。本文通过系统分析架构、问题定义、预训练方案、图像预训练模型使用、损失函数、噪声注入、多步输入、额外静态掩码、多步微调(含大步长模型)以及在更大数据集上训练等设计因素,揭示关键规律。研究涵盖固定网格架构(如UNet、全卷积网络)和网格无关架构(如图神经网络、算子网络)。结果表明,固定网格架构显著优于网格无关架构,提示神经算子等模型需进一步改进。为此,提出结合两者优势的混合系统。此外,多步微调对多数模型性能提升至关重要;预训练目标反而降低表现,而图像预训练模型在某些情况下提供有益归纳偏置。值得注意的是,小模型在大数据集上训练效果优于小数据集长期训练,而大模型主要依赖计算资源增加。这些发现将助力未来更优天气预报系统的设计。

原文摘要 · Abstract (English)

Despite tremendous progress in developing deep-learning-based weather forecasting systems, their design space, including the impact of different design choices, is yet to be well understood. This paper aims to fill this knowledge gap by systematically analyzing these choices including architecture, problem formulation, pretraining scheme, use of image-based pretrained models, loss functions, noise injection, multi-step inputs, additional static masks, multi-step finetuning (including larger stride models), as well as training on a larger dataset. We study fixed-grid architectures such as UNet, fully convolutional architectures, and transformer-based models, along with grid-invariant architectures, including graph-based and operator-based models. Our results show that fixed-grid architectures outperform grid-invariant architectures, indicating a need for further architectural developments in grid-invariant models such as neural operators. We therefore propose a hybrid system that combines the strong performance of fixed-grid models with the flexibility of grid-invariant architectures. We further show that multi-step fine-tuning is essential for most deep-learning models to work well in practice, which has been a common practice in the past. Pretraining objectives degrade performance in comparison to supervised training, while image-based pretrained models provide useful inductive biases in some cases in comparison to training the model from scratch. Interestingly, we see a strong positive effect of using a larger dataset when training a smaller model as compared to training on a smaller dataset for longer. Larger models, on the other hand, primarily benefit from just an increase in the computational budget. We believe that these results will aid in the design of better weather forecasting systems in the future.

天气预报深度学习模型设计架构比较

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。