优化云上遥感数据加载，让GPU跑得更满更快
Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data
- 用贝叶斯优化调优数据读取配置，提升云存储读取效率
- 远程加载吞吐量提升20倍，本地提升4倍，GPU利用率超85%
- 在三个公开遥感数据集上达到本地训练精度，适合大规模遥感建模
在千万亿级遥感数据上训练深度学习模型需将计算与存储分离。但标准PyTorch数据加载器在直接从云存储读取GeoTIFF文件时，难以维持现代GPU的高效利用。本文系统评估了云对象存储与本地SSD的GeoTIFF读取性能，测试不同加载配置与数据参数，聚焦瓦片对齐读取和工作线程池，并采用贝叶斯优化确定各存储类型的最优设置。优化后，远程加载吞吐量相比默认配置提升20倍，本地提升4倍。在三个公开遥感基准上，使用优化远程加载的模型在相同时间预算内达到与本地训练相当的精度，验证指标提升6-15%，且保持85-95%的GPU利用率，而原配置仅为0-30%。代码已开源。
原文摘要 · Abstract (English)
Training deep learning models on petabyte-scale Earth observation (EO) data requires separating compute resources from data storage. However, standard PyTorch data loaders cannot keep modern GPUs utilized when streaming GeoTIFF files directly from cloud storage. In this work, we benchmark GeoTIFF loading throughput from both cloud object storage and local SSD, systematically testing different loader configurations and data parameters. We focus on tile-aligned reads and worker thread pools, using Bayesian optimization to find optimal settings for each storage type. Our optimized configurations increase remote data loading throughput by 20x and local throughput by 4x compared to default settings. On three public EO benchmarks, models trained with optimized remote loading achieve the same accuracy as local training within identical time budgets. We improve validation IoU by 6-15% and maintain 85-95% GPU utilization versus 0-30% with standard configurations. Code is publicly available at https://github.com/microsoft/pytorch-cloud-geotiff-optimization
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。