让显卡通信支持任意位宽，速度最高快3.2倍。
FlashCommunication V2: Bit Splitting and Spike Reserving for Any Bit Communication
- 将任意比特位宽拆解为硬件可支持的基本单元，实现灵活通信。
- 保留极值为浮点数，使2比特量化损失可控，动态范围缩小。
- 适配NVLink和PCIe架构，适合大规模模型训练通信优化。
当前大语言模型的分布式训练与部署中，通信瓶颈已成为关键挑战。本文提出FlashCommunication V2，一种支持任意比特位宽的跨GPU高效通信范式。其核心创新在于提出的比特拆分与尖峰保留技术，解决低比特量化难题。比特拆分将非规则比特位宽分解为基本单元，确保与硬件能力兼容，从而实现任意比特位宽传输。尖峰保留则将数值异常值（即最小值与最大值）以浮点数形式保留，缩小动态数值范围，使量化极限可达2比特且损失可接受。通过精细的软硬件协同设计，该方案在基于NVLink和PCIe的架构上均表现稳健，通信开销显著降低，在AllReduce中最高实现3.2×加速，在All2All中实现2×加速。
原文摘要 · Abstract (English)
Nowadays, communication bottlenecks have emerged as a critical challenge in the distributed training and deployment of large language models (LLMs). This paper introduces FlashCommunication V2, a novel communication paradigm enabling efficient cross-GPU transmission at arbitrary bit widths. Its core innovations lie in the proposed bit splitting and spike reserving techniques, which address the challenges of low-bit quantization. Bit splitting decomposes irregular bit widths into basic units, ensuring compatibility with hardware capabilities and thus enabling transmission at any bit width. Spike reserving, on the other hand, retains numerical outliers (i.e., minima and maxima) as floating-point numbers, which shrinks the dynamic numerical range and pushes the quantization limits to 2-bit with acceptable losses. FlashCommunication V2 significantly enhances the flexibility and resource utilization of communication systems. Through meticulous software-hardware co-design, it delivers robust performance and reduced overhead across both NVLink-based and PCIe-based architectures, achieving a maximum 3.2$\times$ speedup in AllReduce and 2$\times$ in All2All communication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。