arXiv:2503.18617astro-ph.IMastro-ph.SR2025-03中稿 · publication in the…被引 4

用规模定律优化天文光谱模拟器,提升精度与泛化能力。

Scaling Laws for Emulation of Stellar Spectra

  • 基于Transformer和规模定律,系统研究模型、数据、算力的协同增长规律。
  • 算力增10倍时,需数据增2.5倍、模型增3.8倍,才能使误差降低7倍。
  • 为构建通用天文光谱基础模型提供可量化的训练指导,适合天体物理研究者。

基于神经网络的星体参数与元素丰度反演模拟器在现代光谱巡天中日益流行,但其精度和跨域迁移能力常受限制。此前更广的泛化性仅通过大幅增加模型规模实现,如自然语言处理中的Transformer模型。这一现象符合神经网络规模定律:性能随模型规模、计算资源和训练数据量的增加而可预测地提升。本研究证明,该定律同样适用于天文学中的Transformer光谱模拟器。基于我们之前的TransformerPayne工作,并引入自然语言模型中的最大更新参数化技术,我们提出了实现最优性能的训练指南。结果表明,在探索的参数范围内,清晰的规模关系浮现。具体而言,当训练算力增加十倍时,要实现均方误差降低七倍,需约2.5倍的数据量增长和3.8倍的模型规模扩展。本研究为发展具备更强跨域迁移能力的光谱基础模型奠定了基础。

原文摘要 · Abstract (English)

Neural network-based emulators for the inference of stellar parameters and elemental abundances represent an increasingly popular methodology in modern spectroscopic surveys. However, these approaches are often constrained by their emulation precision and domain transfer capabilities. Greater generalizability has previously been achieved only with significantly larger model architectures, as demonstrated by Transformer-based models in natural language processing. This observation aligns with neural scaling laws, where model performance predictably improves with increased model size, computational resources allocated to model training, and training data volume. In this study, we demonstrate that these scaling laws also apply to Transformer-based spectral emulators in astronomy. Building upon our previous work with TransformerPayne and incorporating Maximum Update Parametrization techniques from natural language models, we provide training guidelines for scaling models to achieve optimal performance. Our results show that within the explored parameter space, clear scaling relationships emerge. These findings suggest that optimal computational resource allocation requires balanced scaling. Specifically, given a tenfold increase in training compute, achieving an optimal seven-fold reduction in mean squared error necessitates an approximately 2.5-fold increase in dataset size and a 3.8-fold increase in model size. This study establishes a foundation for developing spectral foundational models with enhanced domain transfer capabilities.

光谱模拟规模定律Transformer天体物理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。