提出流式批处理模型，提升异构环境下训练与推理的效率和容错性。
The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Execution
- 以分区为执行单元，动态流式调度实现弹性扩展。
- 在异构批量推理中吞吐量提升2.5至12倍。
- 适合大规模多模态模型训练，显著优化异构集群性能。
尽管机器学习训练与推理均依赖GPU，但基于CPU的数据处理常成为瓶颈。现有基于批处理或流处理的分布式系统假设资源同质，难以高效利用异构资源，且故障恢复与重配置开销高。本文提出流式批处理模型，融合批处理与流处理优势，以分区为执行单位实现弹性，并支持分区在异构算子间动态创建与流式传输，实现内存高效的流水线。我们构建了Ray Data系统，在异构批量推理中相较传统批处理和流处理系统吞吐量提升2.5至12倍；利用异构集群，相比单节点数据加载器，稳定扩散等多模态模型训练吞吐率提升31%。
原文摘要 · Abstract (English)
While ML model training and inference are both GPU-intensive, CPU-based data processing is often the bottleneck. Distributed data processing systems based on the batch or stream processing models assume homogeneous resource requirements. They excel at CPU-based computation but either under-utilize heterogeneous resources or impose high overheads on failure and reconfiguration. We introduce the streaming batch model, a hybrid of batch and streaming that enables efficient and fault-tolerant heterogeneous execution. The key idea is to use partitions as the unit of execution to achieve elasticity, but to allow partitions to be dynamically created and streamed between heterogeneous operators for memory-efficient pipelining. We present Ray Data, a streaming batch system that improves throughput on heterogeneous batch inference pipelines by 2.5-12$\times$ compared to traditional batch and stream processing systems. By leveraging heterogeneous clusters, Ray Data improves training throughput for multimodal models such as Stable Diffusion by 31% compared to single-node ML data loaders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。