构建跨领域物理模拟基础模型，提升流体类连续体动力学预测能力
Walrus: A Cross-Domain Foundation Model for Continuum Dynamics
- 基于谐波分析稳定长期动态，结合负载均衡分布式训练与自适应分词
- 在19个跨学科场景下预训练，长短期预测均优于现有模型
- 适合从事物理模拟、科学计算的科研与工程人员使用
基础模型已重塑语言与视觉领域的机器学习，但在物理仿真中尚未实现同等影响。数据异质性与不稳定的长期动态阻碍从多样化动力学中学习，而不同分辨率与维度则挑战现代硬件上的高效训练。通过实证与理论分析，我们提出新方法以缓解这些障碍：基于谐波分析的稳定化机制、负载均衡的2D/3D分布式训练策略，以及计算自适应分词。基于这些技术，我们开发了Walrus——一个专为流体类连续体动力学设计的Transformer基础模型。Walrus在涵盖天体物理、地球科学、流变学、等离子体物理、声学及经典流体的19个多样化场景上进行预训练。实验表明，其在下游任务中对短时与长时预测均优于先前基础模型，且跨预训练数据广度表现更优；消融研究证实了所提方法对预测稳定性、训练吞吐量与迁移性能的提升价值。代码与权重已公开供社区使用。
原文摘要 · Abstract (English)
Foundation models have transformed machine learning for language and vision, but achieving comparable impact in physical simulation remains a challenge. Data heterogeneity and unstable long-term dynamics inhibit learning from sufficiently diverse dynamics, while varying resolutions and dimensionalities challenge efficient training on modern hardware. Through empirical and theoretical analysis, we incorporate new approaches to mitigate these obstacles, including a harmonic-analysis-based stabilization method, load-balanced distributed 2D and 3D training strategies, and compute-adaptive tokenization. Using these tools, we develop Walrus, a transformer-based foundation model developed primarily for fluid-like continuum dynamics. Walrus is pretrained on nineteen diverse scenarios spanning astrophysics, geoscience, rheology, plasma physics, acoustics, and classical fluids. Experiments show that Walrus outperforms prior foundation models on both short and long term prediction horizons on downstream tasks and across the breadth of pretraining data, while ablation studies confirm the value of our contributions to forecast stability, training throughput, and transfer performance over conventional approaches. Code and weights are released for community use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。