arXiv:2605.09994cs.DCcs.LG2026-05

为大模型训练设计了原生对象存储的数据平面,实现稳定高吞吐与故障隔离。

BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training

论文配图:BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
图 1 · 摘自论文原文
  • 基于版本化清单和条件写入,实现训练级数据一致性。
  • 在64张GPU上,吞吐高于本地加载器,读延迟低于Kafka。
  • 适合大规模多模态预训练与微调场景的高效数据流水线构建。

现代大基础模型(LFM)训练已将数据管道从静态摄入层转变为需与训练过程协同演进的动态组件。现有系统能力不足:共置数据加载器缺乏故障隔离,消息队列式分离数据加载器使用记录/偏移抽象,无法表达分布式训练所需的批次语义。我们提出BatchWeave,一种面向分布式大模型训练的原生对象存储数据平面。BatchWeave利用版本化清单和条件对象写入,协调批次发布、恢复与生命周期管理。首先,提出事务性全局批次(TGB),在版本化清单的ACID存储语义基础上,扩展了训练特性的强一致性,包括原子全节点批次可见性、全局有序步骤序列、检查点对齐的生命周期管理及端到端精确一次恢复。其次,通过提交协议持久化生产者状态,直接在存储层实现恢复与保留,并以分布式检查点状态控制回收。第三,其去中心化自适应提交(DAC)算法在清单增长时仍保持稳定摄入吞吐,无需生产者间通信。在64张GPU上针对大规模多模态预训练与SFT工作负载的评估显示,BatchWeave在吞吐上优于共置数据加载器并提供完整故障隔离,在摄入吞吐上超越Apache Kafka,消费者读延迟也低于Kafka。

原文摘要 · Abstract (English)

Modern Large Foundation Model (LFM) training has transformed the data pipeline from a static ingestion layer into a dynamic component that must co-evolve with the training process. Existing systems are ill-equipped: colocated dataloaders offer no failure isolation, while message queue-based disaggregated dataloaders operate on a record/offset abstraction that cannot express the batch-level semantics required by distributed training. We present BatchWeave, an object-store-native training data plane for distributed LFM training. BatchWeave uses versioned manifests and conditional object writes to coordinate batch publication, recovery, and lifecycle management. First, it introduces the Transactional Global Batch (TGB), which builds on versioned-manifest ACID storage semantics and extends them with training-specific consistency, including atomic all-rank batch visibility, a globally ordered step sequence, checkpoint-aligned lifecycle management, and end-to-end exactly-once recovery. Second, it realizes recovery and retention directly in the storage layer, by durably persisting producer state through the commit protocol and tying reclamation to distributed checkpoint state. Third, its Decentralized Adaptive Commit (DAC) algorithm sustains stable ingestion throughput as the manifest grows, without any inter-producer communication. Evaluations on large-scale multimodal pre-training and SFT workloads using 64 GPUs show that BatchWeave outperforms colocated dataloader throughput while providing full failure isolation, outperforms Apache Kafka in ingestion throughput, and achieves lower consumer read latency than Kafka.

大模型训练数据流水线对象存储故障隔离

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。