用数据驱动方法预测训练时I/O性能，快速推荐最优存储配置。
Predictive Modeling of I/O Performance for Machine Learning Training Pipelines: A Data-Driven Approach to Storage Optimization
- 基于141组实测数据，用XGBoost建模预测I/O吞吐量。
- 模型平均误差仅11.8%，R²达0.991，可精准定位瓶颈。
- 适合优化机器学习系统存储配置，省去繁琐调参试错。
现代机器学习训练日益受数据I/O限制，而非计算能力。GPU常因等待数据而闲置，利用率低于50%。本文提出一种机器学习方法，用于预测I/O性能并推荐最优存储配置。通过系统性基准测试，在不同存储后端（NVMe SSD、网络附加存储、内存文件系统）、数据格式和访问模式下收集了141组观测数据，涵盖底层I/O操作与完整训练流程。评估七种回归模型和三种分类方法后，XGBoost表现最佳，达到0.991的R²值，平均预测误差为11.8%。特征重要性分析显示，吞吐量指标和批处理大小是主要性能影响因素。该方法可将配置时间从数天的试错缩短至数分钟的预测推荐。该方法可复现且可扩展至其他机器学习系统资源管理问题。代码与数据已公开于https://github.com/knkarthik01/gpu_storage_ml_project。
原文摘要 · Abstract (English)
Modern machine learning training is increasingly bottlenecked by data I/O rather than compute. GPUs often sit idle at below 50% utilization waiting for data. This paper presents a machine learning approach to predict I/O performance and recommend optimal storage configurations for ML training pipelines. We collected 141 observations through systematic benchmarking across different storage backends (NVMe SSD, network-attached storage, in-memory filesystems), data formats, and access patterns, covering both low-level I/O operations and full training pipelines. After evaluating seven regression models and three classification approaches, XGBoost achieved the best performance with R-squared of 0.991, predicting I/O throughput within 11.8% error on average. Feature importance analysis revealed that throughput metrics and batch size are the primary performance drivers. This data-driven approach can reduce configuration time from days of trial-and-error to minutes of predictive recommendation. The methodology is reproducible and extensible to other resource management problems in ML systems. Code and data are available at https://github.com/knkarthik01/gpu_storage_ml_project
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。