基于Transformer的物理模拟代理模型，高效处理多种偏微分方程。
PDE-Transformer: Efficient and Versatile Transformers for Physics Simulations
- 为物理场设计独立时空令牌，通过通道自注意力保持信息密度。
- 在16类偏微分方程数据集上超越计算机视觉最优Transformer模型。
- 预训练模型在下游任务表现更优，适合构建物理科学大模型。
我们提出PDE-Transformer，一种改进的基于Transformer的架构，用于规则网格上的物理模拟代理建模。结合扩散Transformer的最新架构改进与面向大规模模拟的特定调整，构建出更具可扩展性与通用性的通用Transformer架构，可作为物理科学中大型基础模型的骨干。我们在包含16种不同类型偏微分方程的大规模数据集上验证，该架构在性能上超越现有计算机视觉领域的先进Transformer模型。我们提出对不同物理通道分别嵌入为时空令牌，通过通道特异性自注意力进行交互，有助于在同时学习多种偏微分方程时维持一致的信息密度。实验表明，预训练模型在多个挑战性下游任务中表现优于从头训练，并且超越其他物理模拟基础模型架构。
原文摘要 · Abstract (English)
We introduce PDE-Transformer, an improved transformer-based architecture for surrogate modeling of physics simulations on regular grids. We combine recent architectural improvements of diffusion transformers with adjustments specific for large-scale simulations to yield a more scalable and versatile general-purpose transformer architecture, which can be used as the backbone for building large-scale foundation models in physical sciences. We demonstrate that our proposed architecture outperforms state-of-the-art transformer architectures for computer vision on a large dataset of 16 different types of PDEs. We propose to embed different physical channels individually as spatio-temporal tokens, which interact via channel-wise self-attention. This helps to maintain a consistent information density of tokens when learning multiple types of PDEs simultaneously. We demonstrate that our pre-trained models achieve improved performance on several challenging downstream tasks compared to training from scratch and also beat other foundation model architectures for physics simulations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。