XFP动态量化大模型，自动适配质量要求，提升推理速度与内存效率。
XFP: Quality-Targeted Adaptive Codebook Quantization with Sparse Outlier Separation for LLM Inference
- 按通道设定质量阈值，自动确定码本大小和异常值预算，无需校准数据
- 在122B模型上实现138词/秒推理，比Marlin INT4快49%，准确率达94.49%
- 适用于内存受限场景，支持全专家模型压缩，兼顾速度、精度与内存
我们提出XFP，一种面向大模型推理的动态权重量化方法。其反向工作流：由操作员设定每通道余弦相似度的质量下限（注意力与共享专家设严格下限，路由专家设宽松下限），系统自动决定码本大小、异常值预算及每层打包方式——无需海森矩阵、无需校准数据、无需手动选择位宽。每个权重重构为稀疏的fp16异常残差与密集的子字节索引张量，映射至每组自学习码本。两种存储模式共享同一前端与融合解码核：V2（每通道Lloyd）与V2a（每层共享L=32码本库）。在Qwen3.5-122B-A10B上，采用V2模式时，于工作站硬件（RTX PRO 6000 Blackwell，TP=2）实现138 tok/s单流解码，GSM8K严格匹配率94.49%（3种子，n=3957），较Marlin INT4快49%（TP=1）。对于无法容纳于目标内存的模型，提出H-Process：基于质量驱动迭代调整两个余弦阈值，寻找模型恰好适配且输出合理的运行点。三个约束定义搜索空间：操作员设定的阈值、加载时的OOM边界、生成过程中的垃圾边界（余弦相似度引导；基准测试验证）。在Qwen3.5-397B-A17B（每层512个路由专家）上，H-Process将全专家群体压缩至2×96GB，有效比特约3.4，实现100.9 tok/s长输出解码，1319题集上GSM8K严格匹配率66.72%（单种子提交；多种子评估进行中），同时超越带路由专家剪枝的INT4，在内存、吞吐与准确率上全面领先。
原文摘要 · Abstract (English)
We introduce XFP, a dynamic weight quantizer for LLM inference that inverts the conventional workflow: the operator specifies reconstruction quality floors on per-channel cosine similarity (one strict floor for attention and shared experts, one lazy floor for routed-expert MoE); XFP determines codebook size, outlier budget, and packing per layer automatically -- no Hessian, no calibration data, no manual bit-width selection. Each weight matrix is decomposed into a sparse fp16 outlier residual and a dense sub-byte index tensor into a per-group learned codebook. Two storage modes share one auto-select frontend and one fused decode kernel: V2 (per-channel Lloyd) and V2a (shared library of L=32 codebooks per layer). On Qwen3.5-122B-A10B under V2, XFP reaches 138 tok/s single-stream decode on workstation hardware (RTX PRO 6000 Blackwell, TP=2) at 94.49% GSM8K strict-match (3 seeds, n=3957), and is 49% faster than Marlin INT4 at TP=1. For models that do not fit in the target memory envelope, we present the H-Process: a quality-driven iteration over the two cosine thresholds that finds the operating point at which the model just fits while still producing sensible output. Three constraints define its search space: the operator-set thresholds, an OOM boundary at quantize-on-load, and a garbage boundary in generation (cosine similarity steers; benches verify). On Qwen3.5-397B-A17B (512 routed experts/layer), the H-Process fits the full expert population into 2x96 GB at ~3.4 effective bits and delivers 100.9 tok/s long-output decode at 66.72% GSM8K strict-match on the full 1319-problem set (single seed at submission; multi-seed evaluation in progress), exceeding INT4 with routed-expert pruning on memory, throughput, and accuracy simultaneously.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。