用三值权重在CPU上实现无乘法推理，速度提升29.6倍。
FairyFuse: Multiplication-Free LLM Inference on CPUs via Fused Ternary Kernels
- 将线性层的8个子矩阵融合为单个AVX-512循环，用掩码加减替代乘法。
- 在英特尔至强处理器上达到每秒32.4个词元，比llama.cpp快1.24倍。
- 适合在无GPU的低端设备上部署低精度大模型，性能接近浮点原版。
大型语言模型越来越多地部署在仅含CPU的平台，内存带宽成为自回归生成的主要瓶颈。将权重量化至4比特或以下可缓解内存压力，但现有系统仍需反量化并执行浮点乘法，限制了性能提升。{-1, 0, +1}三值权重提供更高效替代方案，可用条件加、减或无操作替代乘法。尽管Fairy2i表明三值大模型可达到FP16质量，其运行时未充分利用该结构。我们提出FairyFuse，一种推理系统，通过将每个广泛线性层的八个实值子GEMV融合为单个AVX-512循环，使用掩码加减实现商品级CPU上的无乘法执行，零浮点乘法。屋顶分析显示，16倍权重压缩使内存受限的CPU上的GEMV从内存瓶颈转向计算主导，带来29.6倍内核加速，对GPU则收益甚微。端到端测试中，FairyFuse在单颗Intel Xeon 8558P上实现每秒32.4词元，优于llama.cpp Q4_K_M的1.24倍，且近似无损：WikiText-2困惑度5.52（对比FP16的5.47），下游准确率66.0%。
原文摘要 · Abstract (English)
Large language models are increasingly deployed on CPU-only platforms where memory bandwidth is the primary bottleneck for autoregressive generation. Weight quantization to four bits or below reduces memory pressure, yet existing systems still dequantize weights and perform floating-point multiplications, limiting the achievable gains. Ternary weights in {-1, 0, +1} provide a more efficient alternative, replacing multiplications with conditional additions, subtractions, or no-ops. While Fairy2i shows that ternary LLMs can match FP16 quality, its runtime does not exploit this structure. We present FairyFuse, an inference system that enables multiplication-free execution on commodity CPUs by fusing the eight real-valued sub-GEMVs of each widely-linear layer into a single AVX-512 loop using masked additions and subtractions, with zero floating-point multiplications. Roofline analysis shows that 16x weight compression shifts memory-bound GEMV toward the compute regime on bandwidth-limited CPUs, yielding a 29.6x kernel speedup while offering little benefit on GPUs. End-to-end, FairyFuse achieves 32.4 tokens per second on a single Intel Xeon 8558P, outperforming llama.cpp Q4_K_M by 1.24x with near-lossless quality (WikiText-2 perplexity 5.52 vs. 5.47 FP16; downstream accuracy 66.0%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。