让大模型训练动态分批更高效，实时计算样本成本避免资源浪费。
Online Dynamic Batching with Formal Guarantees for LLM Training
- 训练时实时观测样本处理开销，动态组批提升效率。
- 单机全量微调下吞吐最高提升3.78倍，质量与固定批次相当。
- 无需改模型或内核，适合高异构数据的在线微调场景。
现代大模型训练打破了离线分批采样的核心假设：样本的真实训练成本只有在完成预处理、增强、模板化、分词及多模态视觉令牌扩展后才能确定。若不预先构建依赖预处理和增强的长度缓存，分批过程便无法感知决定填充、内存占用和GPU饱和度的关键指标。本文提出在线动态分批（ODB），一种无需修改模型、优化器或注意力内核的DataLoader侧即插即用系统，在可准确观测成本的时刻进行批生成，同时保持分布式数据并行（DDP）步对齐。我们形式化这一同步要求为分布式组对齐问题（DGAP），并证明在默认连接模式身份覆盖和可选非连接样本配额封闭下，系统具备无死锁且有界终止的保证。在UltraChat/LLaVA/ShareGPT4o等公开2B/8B Qwen3-VL数据集上的实验表明，与固定批次标准方法相比，ODB在单节点全量微调/LoRA中实现1.58–2.51倍的原始样本吞吐提升，在双节点全量微调中达1.71–3.78倍，且性能相当；生产级多模态混合数据达到4.43倍。相较于GMT/BMT等离线令牌预算调度器，ODB在UltraChat/LLaVA上差距小于15%，在高视觉复杂度的ShareGPT4o上更快，单节点提升2.24–2.39倍，双节点达3.06–3.69倍。总体而言，ODB填补了高异构大模型微调中在线、即插即用、有形式保障的分批方案空白，实现高吞吐、高质量、无预计算缓存和无需内核重写。
原文摘要 · Abstract (English)
Modern LLM training breaks a core assumption behind offline batch samplers: the true training cost of a sample is only observable after preprocessing, augmentation, templating, tokenization, and multimodal visual-token expansion. Unless one pays for a preprocessing- and augmentation-dependent length cache, batch construction is therefore blind to the quantity that determines padding, memory use, and GPU saturation. We introduce Online Dynamic Batching (ODB), a DataLoader-side drop-in system that moves batch formation to this point of accurate observability while preserving DDP step alignment. We formalize this synchronization requirement as the Distributed Group Alignment Problem and prove deadlock-free bounded termination with default join-mode identity coverage and opt-in non-join sample-quota closure. ODB requires no model, optimizer, or attention-kernel changes and is released as online-dynamic-batching with lightweight trainer adapters. Across public 2B/8B Qwen3-VL runs on UltraChat/LLaVA/ShareGPT4o, ODB improves literal emitted-sample throughput vs. fixed-batch Standard by 1.58-2.51x on single-node Full FT/LoRA and 1.71-3.78x on two-node Full FT, with Standard-comparable quality; production MM-Mix reaches 4.43x. Against GMT/BMT offline token-budget oracles, ODB is within 15% on UltraChat/LLaVA and faster on high-CV ShareGPT4o: 2.24-2.39x single-node Full FT/LoRA and 3.06-3.69x two-node Full FT. Together, ODB occupies the online/drop-in regime for high-heterogeneity LLM fine-tuning: large throughput gains at Standard-comparable quality, formal DGAP guarantees, and no length-cache precompute or kernel rewrites.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。