Seneca优化多任务训练数据加载，提速近一半
Preparation Meets Opportunity: Enhancing Data Preprocessing for ML Training With Seneca
- 基于性能模型动态分配缓存，区分编码、解码、增强三类数据
- 随机采样时优先使用缓存数据，实现并发任务间资源共享
- 在PyTorch上部署后，训练完成时间减少45.23%，吞吐提升3.45倍
在现代系统中并行训练多媒体机器学习模型时，输入数据预处理常成为瓶颈。为缓解这一问题并缩短并发任务的训练时间，本文提出Seneca——一个优化数据存储与摄入（DSI）管道缓存分区与数据采样的数据加载系统。Seneca采用两项关键技术：首先，基于数据流水线性能模型，对编码、解码和增强三种形式的数据进行最优缓存分区；其次，在随机批次采样过程中，主动优先服务已缓存数据而非未缓存数据，使并发任务间实现资源互惠。我们在PyTorch基础上实现了Seneca，并通过对比多种先进缓存系统验证其有效性。结果表明，相较于原生PyTorch，Seneca将任务完成时间（makespan）降低45.23%；相比当前最佳数据加载器，数据处理吞吐量最高提升3.45倍。
原文摘要 · Abstract (English)
Input data preprocessing is a common bottleneck when concurrently training multimedia machine learning (ML) models in modern systems. To alleviate these bottlenecks and reduce the training time for concurrent jobs, we present Seneca, a data loading system that optimizes cache partitioning and data sampling for the data storage and ingestion (DSI) pipeline. The design of Seneca contains two key techniques. First, Seneca uses a performance model for the data pipeline to optimally partition the cache for three different forms of data (encoded, decoded, and augmented). Second, Seneca opportunistically serves cached data over uncached ones during random batch sampling so that concurrent jobs benefit from each other. We implement Seneca by modifying PyTorch and demonstrate its effectiveness by comparing it against several state-of-the-art caching systems for DNN training. Seneca reduces the makespan by 45.23% compared to PyTorch and increases data processing throughput by up to 3.45x compared to the next best dataloader.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。