用无损压缩解决训练推理中的显存瓶颈，提升效率。
Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended
- 提出无损压缩算法IBP，通过消除张量不变位减少传输数据量。
- 在图神经网络训练中平均提速74%，推荐系统嵌入查询提速180%。
- 支持现有框架无缝集成，适合高吞吐场景的模型部署者使用。
机器学习训练与推理常需处理远超显存容量的数据集，被迫依赖PCIe进行按需张量传输，造成严重传输瓶颈。已有方法采用有损压缩缓解此问题，但引入依赖任务的精度损失，使现有部署复杂甚至不可行。本文探索无损压缩作为替代方案，避免部署复杂性。我们识别出无损压缩可融入ML流水线的位置，同时最小化对GPU执行的干扰。基于研究结果，提出新型无损压缩算法Invariant Bit Packing（IBP），通过识别并消除张量组中的不变位，结合面向GPU优化的解压机制——利用线程束并行、低开销位操作及异步PCIe传输，显著降低数据传输时间。我们提供易用API，已集成至GNN训练、DLRM和LLM推理框架。实验表明，IBP在图神经网络训练中平均提速74%，在DLRM嵌入查找中提速180%，在大模型推理中提速24%。
原文摘要 · Abstract (English)
Machine learning (ML) training and inference often process data sets far exceeding GPU memory capacity, forcing them to rely on PCIe for on-demand tensor transfers, causing critical transfer bottlenecks. Lossy compression has been proposed to relieve bottlenecks but introduces workload-dependent accuracy loss, making it complex or even prohibitive to use in existing ML deployments. We explore lossless compression as an alternative that avoids this deployment complexity. We identify where lossless compression can be integrated into ML pipelines while minimizing interference with GPU execution. Based on our findings, we introduce Invariant Bit Packing (IBP), a novel lossless compression algorithm designed to minimize data transfer time for ML. IBP identifies and eliminates invariant bits across groups of tensors, improving throughput through GPU-optimized decompression that leverages warp parallelism, low-overhead bit operations, and asynchronous PCIe transfers. We provide easy-to-use APIs, showcasing them by adding IBP support to GNN training, as well as DLRM and LLM inference frameworks. IBP achieves, on average, 74% faster GNN training, 180% faster DLRM embedding lookup, and 24% faster LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。