提出细粒度协程模型，显著提升大规模批量推理的效率与资源利用率。
BatchGen: An Architecture for Scalable and Efficient Batch Inference
- 将每个序列视为事件驱动的协程，实现动态任务重排。
- 128卡集群下批处理完成时间缩短2.3倍，内存受限设备上性能提升9.6倍。
- 适合需要高吞吐、低延迟的生产级批量推理场景。
批量推理已成为AI计算的核心模式，但现有推理引擎仍基于交互式服务设计。当扩展至百万级序列时，批处理工作负载暴露出两大根本需求：运行时才显现的极端序列间与序列内负载差异的处理能力，以及在大规模GPU集群中维持高利用率的能力。现有系统难以满足这些需求，导致大量可实现吞吐损失。本文提出面向批量推理的新架构基础——序列协程计算模型，将每个序列表示为细粒度、事件驱动的协程。该模型提供丰富原语，使运行时可动态重组任务，支持更大规模专家级批处理，缓解拖后腿现象，跨设备重新分配任务，并在低成本或内存受限的GPU上保持高利用率。基于此抽象，我们实现了生产就绪的BatchGen系统，在128张GPU集群上，批处理完成时间最多缩短2.3倍；在内存受限加速器上，性能优于最强的卸载基线达9.6倍。项目将开源于https://github.com/batchgen-project/batchgen。
原文摘要 · Abstract (English)
Batch inference has become a central mode of AI computation, yet existing inference engines still rely on execution models designed for interactive serving. When scaled to millions of sequences, batch workloads reveal two fundamental requirements: the ability to handle extreme inter- and intra-sequence load variation that emerges only at runtime, and the ability to sustain high utilization across large fleets of GPUs. Existing systems fail to meet these requirements, losing substantial fractions of achievable throughput. We introduce a new architectural foundation for batch inference: the sequence coroutine compute model, which represents each sequence as a fine-grained, event-driven coroutine. This model exposes expressive primitives that allow the runtime to reorganize work dynamically, enabling larger expert-level batches, mitigating stragglers, reallocating work across devices, and maintaining utilization even on cost-effective or memory-constrained GPUs. Building on this abstraction, we implement BatchGen, a production-ready system that uses the coroutine model at cluster scale. On a 128-GPU cluster, BatchGen reduces batch completion time by up to $2.3\times$, and on memory-constrained accelerators it outperforms the strongest offloading baseline by up to $9.6\times$. We will open-source BatchGen at https://github.com/batchgen-project/batchgen
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。