为边缘CPU设计可伸缩的细粒度量化框架，实现高效低比特大模型推理。
PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference
- 根据用户设定平均位宽，按通道分配2-16位，实现激活感知的混合精度量化。
- 编译时重排与聚类通道，生成兼容SIMD和查表的内核，减少运行时开销70.8%。
- 在3比特下困惑度提升2.4%-32.1%，跨设备性能稳定，能耗增益低于2%。
CPU是设备端大模型推理最通用的目标，但现有低比特量化方法要么精度粗糙，要么细粒度混合精度难以在CPU上高效执行。本文提出PolyQ，一种面向CPU的编译器与量化协同设计框架，在用户指定平均位宽条件下，实现激活感知的通道级比特分配（取值{2,3,4,8,16}）。通过编译时对通道进行重排与聚类，形成比特同质块，生成兼容SIMD与查表的内核，并合并跨算子的可复用重排，使布局规整化脱离运行时路径。该方法将细粒度比特预算适配转化为可在纯CPU上落地的分数比特部署方案。在Falcon-H1-3B、Llama2-13B和Qwen3-32B模型上，基于WikiText-2数据集测试显示，PolyQ在3–6比特范围内保持稳定质量提升，在3比特目标下困惑度较先前方法降低2.4%–32.1%。在三类典型CPU（工作站、笔记本、移动）上的端到端测量表明，编译器布局规整化使激活重排流量降低最高达70.8%，预填充延迟与解码吞吐量几乎随配置比特预算线性增长，每字节能耗开销相对优化的查表后端低于2%。结果表明，分数比特的纯CPU部署在多种边缘设备上均具备实用性、可预测性与能效优势。
原文摘要 · Abstract (English)
CPUs are the most universal target for on-device LLM inference, but existing low-bit quantization methods offer either coarse operating points or fine-grained mixed precision that is difficult to execute efficiently on CPUs. We present PolyQ, a CPU-oriented compiler/quantization co-design for activation-aware channel-wise bit allocation under a user-specified average-bit budget. PolyQ assigns per-channel bit-widths from $\{2,3,4,8,16\}$, then uses a compile-time model compiler to permute and cluster channels into bit-homogeneous blocks, generate SIMD- and LUT-compatible kernels, and merge compatible permutations across operators to keep layout regularization off the runtime path. This turns fine-grained budget fitting into a practical fractional-bit deployment method for CPU-only inference. Across Falcon-H1-3B, Llama2-13B, and Qwen3-32B on WikiText-2, PolyQ provides stable quality scaling from 3--6\,b and improves perplexity by 2.4--32.1\% over prior methods at a 3\,b target. End-to-end measurements on three representative CPUs -- workstation, laptop, and mobile -- show that compiler layout regularization reduces activation reorder traffic by up to 70.8\%, prefill latency and decode throughput scale nearly proportionally with the configured bit budget, and energy/token overhead stays below 2\% relative to an optimized LUT-based back-end. These results show that fractional-bit CPU deployment is practical, predictable, and energy-efficient across diverse edge targets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。