提出新型编码方案，提升大模型通信压缩速度与硬件友好性。
Quad Length Codes for Lossless Compression of e4m3
- 用3位前缀分8区，每区不同码长，256符号直接查表编码
- 对e4m3数据压缩率13.9%，略低于哈夫曼的15.9%但更快更省硬件
- 适合需要高速解码的分布式大模型训练场景
大型语言模型的训练与推理严重依赖并行化和集体通信操作，常受网络带宽瓶颈制约。无损压缩如哈夫曼编码可缓解此问题，但其解码慢且硬件复杂度高，因需深度树遍历。通用编码如指数戈洛姆码解码快，却无法利用符号频率分布。本文提出四长度码（Quad Length Codes），一种兼顾压缩效率与解码速度的混合方案。该编码使用3位前缀将256个符号划分为8个区域，每个区域具有不同的码长和符号数量。通过包含256个条目的查找表实现编码，显著降低硬件实现复杂度，相比哈夫曼树大幅简化。该方案可适配不同符号分布。在e4m3数据类型上，压缩率达13.9%，虽略低于哈夫曼码的15.9%，但显著提升解码速度并降低硬件开销。
原文摘要 · Abstract (English)
Training and serving Large Language Models (LLMs) relies heavily on parallelization and collective operations, which are frequently bottlenecked by network bandwidth. Lossless compression using e.g., Huffman codes can alleviate the issue, however, Huffman codes suffer from slow, bit-sequential decoding and high hardware complexity due to deep tree traversals. Universal codes e.g., Exponential-Golomb codes are faster to decode but do not exploit the symbol frequency distributions. To address these limitations, this paper introduces Quad Length Codes, a hybrid approach designed to balance compression efficiency with decoding speed. The coding scheme uses 3 prefix bits to divide the 256 symbols into 8 areas. Each area has a different code length and encodes a different number of symbols. The scheme uses a Look Up Table with 256 entries, significantly simplifying the hardware implementation compared to Huffman trees. The coding scheme can be adapted for different distributions. For the e4m3 data type, the scheme achieves a compressibility of 13.9% in comparison to 15.9% achieved by Huffman codes, but it significantly speeds up the decoding and simplifies the hardware complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。