FASQ让大模型压缩更灵活高效，无需校准就能在消费级显卡上加速推理。
FASQ: Flexible Accelerated Subspace Quantization for Calibration-Free LLM Compression
- 用子向量和码本参数调节，实现连续精度-大小权衡
- 37%-42%模型大小下准确率超4比特方案，推理速度达FP16的1.8倍
- 支持无校准压缩，适合追求极致推理效率的部署场景
将大语言模型（LLM）压缩至通用GPU部署仍具挑战：传统标量量化仅支持固定位宽（如8/4/3比特），压缩点稀疏且通常需校准数据。本文提出FASQ（柔性加速子空间量化），对LLM权重矩阵应用产品量化。通过调整子向量大小与码本基数两个参数，FASQ构建出覆盖原FP16模型27%-49%大小的连续设计空间，填补固定比特方案的压缩空白。在Meta-Llama-3-8B上，FASQ在37%-42%模型大小下平均准确率达67.1-67.7，优于4比特GPTQ与AWQ，并在Qwen3-8B与Qwen3.5-9B-Base上保持一致性能。为实现推理时产品量化实用化，设计定制CUDA核函数：免查表直接计算的GEMV用于解码，输出驻留双缓冲查表GEMM用于预填充，均支持split-K并行。在RTX 3090上，FASQ实现45.2 tok/s解码（有效4比特，内存减少2.56倍）与51.8 tok/s（有效3比特，内存减少2.80倍），均超越FP16张量核心性能（43.9 tok/s），吞吐量达AWQ的1.6-1.8倍、GPTQ的2.5倍、RTN的4.3-5倍。FASQ是唯一在单张消费级显卡上实现解码加速超过原生FP16的压缩方法，兼具无校准、连续压缩、实时推理优势。
原文摘要 · Abstract (English)
Compressing large language models (LLMs) for deployment on commodity GPUs remains challenging: conventional scalar quantization is limited to fixed bit-widths (e.g., 8/4/3-bit), offers only a few discrete compression points, and typically requires calibration data. We present FASQ (Flexible Accelerated Subspace Quantization), a calibration-free framework that applies product quantization to LLM weight matrices. By tuning two parameters, sub-vector size and codebook cardinality, FASQ exposes a continuous design space spanning 27-49% of the original FP16 model size, filling compression gaps that fixed-bit schemes cannot reach. On Meta-Llama-3-8B, FASQ surpasses 4-bit GPTQ and AWQ in accuracy (67.1-67.7 avg.) at 37-42% model size, with consistent results on Qwen3-8B and Qwen3.5-9B-Base. To make product quantization practical at inference time, we design custom CUDA kernels: a LUT-free direct-compute GEMV for decode and an output-stationary double-buffered LUT GEMM for prefill, both with split-K parallelism. On an RTX~3090, FASQ achieves 45.2 tok/s decode at effective 4-bit (2.56x memory reduction) and 51.8 tok/s at effective 3-bit (2.80x), both surpassing FP16 tensor-core performance (43.9 tok/s) and delivering 1.6 to 1.8x the throughput of AWQ, 2.5 to 2.5x of GPTQ, and 4.3 to 5x of RTN. FASQ is the only compressed method that accelerates decode beyond FP16, offering calibration-free compression, continuous size-quality trade-offs, and real-time inference on a single consumer GPU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。