用深度学习提升科学数据压缩,发现预测精度高不等于压缩比高。
Can Deep Neural Networks Improve Compression of Very Large Scientific Data?

- 将时序深度模型融入传统压缩流程,实现自回归预测
- 对可预测变量压缩比最高提升9.6倍,重建质量提高91%
- 压缩效果受残差空间结构影响,仅靠高精度预测不够
误差有界有损压缩是管理现代模拟与观测设备产生的海量科学数据的核心技术。当前主流压缩器采用预测-残差范式,压缩效果取决于预测器的准确性:更精准的预测产生更小的残差,更容易压缩。这引发一个问题:现代机器学习模型能否作为科学数据压缩的更优预测器?直接验证此问题困难,因开发专用ML预测器需大量资源。为此,我们利用气候领域已存在高精度预训练气象预报基础模型的优势,构建了一个集成时空深度学习模型的压缩框架。该框架支持自回归预测且避免误差累积。以ERA5气候数据(约1.7TB)为基准,对比三种不同ML预测器:基于VAEformer的编码器(CRA5)、图神经网络预报器(GraphCast)、视觉变换器预报器(Aurora),与最新压缩器SZ3.1在相同量化和熵编码后端下进行评估。结果显示:尽管ML预测器显著提升预测精度,使重建质量最高提升91%,部分变量压缩比达9.6倍,但整体数据集压缩比未改善。原因在于:残差的空间结构对熵编码效率起决定性作用,单纯提高预测精度不足以提升压缩性能。
原文摘要 · Abstract (English)
Error-bounded lossy compression is a fundamental technique for managing the rapidly growing volumes of scientific data produced by modern simulations and observational instruments. Most state-of-the-art-compressors follow a prediction-residual paradigm, where compression effectiveness depends on the quality of the predictor: more accurate predictions generate smaller residuals that are easier to compress. This observation raises a question: can modern machine learning models serve as superior predictors for scientific data compression? Answering this question directly is challenging because developing compression-specific ML predictors requires substantial resources. Instead, we leverage the climate domain where highly accurate pretrained weather forecasting foundation models already exist, making them an ideal testbed. We present a framework that integrates spatial and temporal deep learning models into a conventional error-bounded compression pipeline. The framework supports auto-regressive forecasting models and avoids error accumulation. Using ERA5 climate data as a representative large-scale scientific dataset, we evaluate three distinct ML predictors: a VAEformer-based codec (CRA5), a graph neural network forecaster (GraphCast), and a vision-transformer forecaster (Aurora), against the state-of-the-art compressor SZ3.1 under identical quantization and entropy-coding backends. Our evaluation over approximately 1.7 TB of data reveals a surprising result: although ML predictors generate more accurate predictions and can improve reconstruction quality by up to 91% while achieving up to 9.6x higher compression ratios for highly predictable variables, they do not improve overall dataset-level compression ratio. We show that prediction accuracy alone is insufficient: the spatial structure of the resulting residuals plays a decisive role in entropy coding efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。