让大模型推理更快：通过动态分离稀疏与密集部分，实现FP4量化下的高效计算。
SharQ: Bridging Activation Sparsity and FP4 Quantization for LLM Inference
- 动态生成稀疏掩码，将激活值分为异常值主导的稀疏主干和残差部分。
- 在语言和视觉任务中恢复43%~63%的精度损失，实测加速2.2~2.4倍。
- 无需训练或调参，适配多种低精度格式，适合部署在现代GPU上。
低比特浮点格式与半结构化稀疏性日益被现代加速器支持，但将其联合用于大模型激活压缩仍具挑战：激活值中的输入相关异常值会主导FP4量化的块尺度，直接应用N:M稀疏掩码则会丢弃中等数值，导致稀疏化损失与量化误差耦合。本文提出SharQ，一种无需训练的推理方法，通过在线稀疏-密集分解桥接激活稀疏性与FP4量化。对每个激活张量，SharQ生成输入自适应的N:M掩码，提取以异常值为主的稀疏主干并量化为FP4，同时定义相对于量化后稀疏主干的密集残差。稀疏路径使用FP4 GEMM处理主干，密集路径则补偿掩码引入的激活损失与稀疏路径的量化误差。两条路径共享单一FP4权重载荷，采用路径特异的尺度视图，并由融合准备核将掩码生成、残差构建与层归一化合并为一个算子。SharQ无需校准数据、重训练或模型特定调参。在Llama-3.1-8B、Qwen2.5-7B、Qwen3-30B-A3B和Qwen3-VL-8B上评估，其在语言与视觉语言任务中恢复了43%~63%的NVFP4到FP16精度差距，并泛化至NVFP4、HiF4与MXFP4格式。在RTX 5090上，相比FP16降低2.2~2.4倍延迟，相比FP8提升1.2~1.4倍吞吐；与SageAttention结合时，在Wan2.2-T2V-A14B视频生成中实现最高1.58倍加速。代码已开源。
原文摘要 · Abstract (English)
Low-bit floating-point formats and semi-structured sparsity are increasingly supported by modern accelerators, yet combining them for LLM activation compression remains challenging: activations contain input-dependent outliers that dominate block scales in FP4 quantization, and directly applying N:M sparsity masks discards moderate values, coupling sparsification loss with quantization error. We introduce SharQ, a training-free inference method that bridges activation sparsity and FP4 quantization through an online sparse--dense decomposition. For each activation tensor, SharQ generates an input-adaptive N:M mask to extract an outlier-dominated sparse backbone, quantizes it to FP4, and defines a dense residual relative to the quantized sparse backbone rather than the unquantized sparse values. A sparse FP4 GEMM processes the backbone while a dense FP4 GEMM compensates for both mask-induced activation loss and sparse-path quantization error. The two paths share a single FP4 weight payload with path-specific scale views, and a fused preparation kernel absorbs mask generation, residual construction, and layer normalization into one operator. SharQ requires no calibration data, retraining, or model-specific tuning. Evaluated on Llama-3.1-8B, Qwen2.5-7B, Qwen3-30B-A3B, and Qwen3-VL-8B, SharQ recovers 43--63% of the NVFP4-to-FP16 accuracy gap across language and vision-language tasks, and generalizes across NVFP4, HiF4, and MXFP4 formats. On an RTX 5090, SharQ delivers 2.2--2.4$\times$ latency reduction over FP16 and 1.2--1.4$\times$ throughput improvement over FP8 in language model serving, and up to 1.58$\times$ speedup on Wan2.2-T2V-A14B video generation when combined with SageAttention. Our code is available at https://github.com/actypedef/SharQ.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。