用连续深度神经微分方程改进Transformer,让天气预报更准确且符合物理规律。
PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting

- 在Transformer编码器中引入神经微分方程,实现连续深度更新。
- 通过物理信息损失函数软约束,使预测结果符合气象基本规律。
- 适合需要高精度短期天气预报的研究者与应用团队。
业务化天气预报长期依赖基于物理的数值天气预报(NWP),其高精度伴随巨大计算开销和复杂的模拟流程。近期基于Transformer的预报模型提供了高效的数据驱动替代方案,但这些模型缺乏物理感知能力。此外,标准Transformer编码器通过离散层更新表示,可能不适用于建模平滑的潜在动态。本文提出一种连续深度Transformer编码器,将神经常微分方程(Neural ODE)动态嵌入每个编码器块中,以自适应数值积分替代离散残差更新。同时引入双分支注意力模块,结合传统的补丁级自注意力与一个对注意力逻辑施加导数算子的辅助分支,提供额外的敏感变化交互信号。为进一步对齐预测与物理原理,提出定制化的物理信息训练目标,将物理一致性作为软约束。在标准离散Transformer基线及现有连续时间神经微分方程预报变体上进行评估,验证了PINN-Cast在短时天气预报中的有效性。
原文摘要 · Abstract (English)
Operational weather prediction has long relied on physics-based numerical weather prediction (NWP), whose accuracy comes at the cost of substantial compute and complex simulation workflows. Recent transformer-based forecasters offer efficient data-driven alternatives, however transformers are physics-agnostic models. Additionally, standard transformer encoders evolve representations through discrete layer updates that may be less suited to modeling smooth latent dynamics. In this work, we propose a continuous-depth transformer encoder for weather forecasting that integrates Neural Ordinary Differential Equation (Neural ODE) dynamics within each encoder block. Specifically, we replace discrete residual updates with ODE-based updates solved using adaptive numerical integration. We also introduce a two-branch attention module that combines conventional patch-wise self-attention with an auxiliary branch that applies a derivative operator to attention logits, providing an additional change-sensitive interaction signal. To further align forecasts with governing principles, we propose a customized physics-informed training objective that enforces physical consistency as a soft constraint. We evaluate the proposed method against a standard discrete transformer baseline and an existing continuous-time Neural ODE forecasting variant, demonstrating the importance of PINN-Cast in short term weather forecasting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。