用压缩技术降低气象数据存储量，提升大模型效率
Compressing high-resolution data through latent representation encoding for downscaling large-scale AI weather forecast model
- 设计变分自编码器压缩1公里分辨率气象数据
- 3年数据从8.61TB压缩至204GB，信息保留完整
- 压缩后数据可直接用于模型训练，效果不降
人工智能在气象研究中的快速发展得益于对大规模高维数据的学习能力，但随之而来的是数据处理成本高昂与计算资源受限的挑战。受计算机视觉中神经图像压缩（NIC）的启发，本研究提出一种专为高分辨率气象数据设计的变分自编码器（VAE）框架，以压缩中国气象局陆面数据同化系统（HRCLDAS）的1公里空间分辨率数据。该框架将3年HRCLDAS数据的存储量从8.61 TB缩减至204 GB，同时有效保留关键信息。此外，在降尺度任务中，基于压缩数据训练的模型表现与原始数据训练的模型相当。结果表明，压缩后的数据在后续气象研究中具有高效性和应用潜力。
原文摘要 · Abstract (English)
The rapid advancement of artificial intelligence (AI) in weather research has been driven by the ability to learn from large, high-dimensional datasets. However, this progress also poses significant challenges, particularly regarding the substantial costs associated with processing extensive data and the limitations of computational resources. Inspired by the Neural Image Compression (NIC) task in computer vision, this study seeks to compress weather data to address these challenges and enhance the efficiency of downstream applications. Specifically, we propose a variational autoencoder (VAE) framework tailored for compressing high-resolution datasets, specifically the High Resolution China Meteorological Administration Land Data Assimilation System (HRCLDAS) with a spatial resolution of 1 km. Our framework successfully reduced the storage size of 3 years of HRCLDAS data from 8.61 TB to just 204 GB, while preserving essential information. In addition, we demonstrated the utility of the compressed data through a downscaling task, where the model trained on the compressed dataset achieved accuracy comparable to that of the model trained on the original data. These results highlight the effectiveness and potential of the compressed data for future weather research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。