针对MIG推理服务器的预处理瓶颈，提出软硬件协同方案提升性能。
PREBA: A Hardware/Software Co-Design for Multi-Instance GPU based AI Inference Servers
- 用FPGA加速数据预处理，释放MIG算力潜力。
- 动态批处理系统使吞吐量提升3.7倍，尾延迟降低3.4倍。
- 适合构建高能效、低成本的AI推理服务系统。
NVIDIA的多实例GPU（MIG）可将单个大GPU分割为多个小计算切片。本文研究MIG在AI推理服务器中的表现，发现数据预处理阶段成为主要性能瓶颈。为此提出PREBA，一种面向MIG推理服务器的软硬件协同设计：首先采用FPGA实现专用数据预处理加速，释放计算资源；再通过动态批处理系统进一步提升性能。该方案在真实系统中端到端实现，使吞吐量提升3.7倍，尾延迟降低3.4倍，能效提升3.5倍，成本效率提高3.0倍。
原文摘要 · Abstract (English)
NVIDIA's Multi-Instance GPU (MIG) is a feature that enables system designers to reconfigure one large GPU into multiple smaller GPU slices. This work characterizes this emerging GPU and evaluates its effectiveness in designing high-performance AI inference servers. Our study reveals that the data preprocessing stage of AI inference causes significant performance bottlenecks to MIG. To this end, we present PREBA, which is a hardware/software co-design targeting MIG inference servers. Our first proposition is an FPGA-based data preprocessing accelerator that unlocks the full potential of MIG with domain-specific acceleration of data preprocessing. The MIG inference server unleashed from preprocessing overheads is then augmented with our dynamic batching system that enables high-performance inference. PREBA is implemented end-to-end in real systems, providing a 3.7x improvement in throughput, 3.4x reduction in tail latency, 3.5x improvement in energy-efficiency, and 3.0x improvement in cost-efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。