用简单模型和持续训练，发现气象模拟的神经尺度规律。
On Neural Scaling Laws for Weather Emulation through Continual Training
- 采用最小化Swin Transformer与恒定学习率持续训练。
- 冷却阶段可提升多步预报精度与谱损失优化效果。
- 找到计算最优训练方案，助力资源高效分配。
神经尺度定律在自然语言处理与计算机视觉中是构建基础模型的核心，本研究将其拓展至科学机器学习领域，聚焦气象预报模型。为简化分析，采用通用且可扩展的Swin Transformer架构，并使用恒定学习率结合周期性冷却的持续训练策略。实验表明，该方法能实现可预测的规模增长趋势，甚至优于标准余弦退火调度。冷却阶段可重新利用以提升下游性能，如更长预报时长的多步滚动预测及通过谱损失调整获得更清晰的预测结果。系统探索了不同模型与数据规模在多种计算预算下的表现，构建了IsoFLOP曲线，识别出计算最优训练策略。外推至更大规模揭示潜在性能极限，证明神经尺度定律可作为高效资源分配的重要诊断工具。代码已开源以支持复现。
原文摘要 · Abstract (English)
Neural scaling laws, which in some domains can predict the performance of large neural networks as a function of model, data, and compute scale, are the cornerstone of building foundation models in Natural Language Processing and Computer Vision. We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting. To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture, and we use continual training with constant learning rates and periodic cooldowns as an efficient training strategy. We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules. Cooldown phases can be re-purposed to improve downstream performance, e.g., enabling accurate multi-step rollouts over longer forecast horizons as well as sharper predictions through spectral loss adjustments. We also systematically explore a wide range of model and dataset sizes under various compute budgets to construct IsoFLOP curves, and we identify compute-optimal training regimes. Extrapolating these trends to larger scales highlights potential performance limits, demonstrating that neural scaling can serve as an important diagnostic for efficient resource allocation. We open-source our code for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。