INT8量化推理中,不同内核导致输出不一致,根源在缩放与舍入阶段。
The Integer Alibi: Localizing Cross-Kernel Divergence in INT8-Quantized LLM Inference
- 通过固定参数仅替换线性内核,发现CUTLASS与Triton在相同输入下产生差异。
- 在2的幂次缩放下输出完全一致,真实缩放下差异≤1个bfloat16间隔。
- 可定位分歧源为缩放和输出舍入,适合模型部署与内核验证场景。
两个实现相同缩放INT8 GEMM接口的GPU内核通常被视为可互换。我们验证此假设:在固定检查点、提示、硬件、推理引擎、解码方式及量化配置的前提下,仅在vLLM中替换INT8线性内核(CUTLASS vs Triton)。每个1.7B模型在冷启动后能自复现比特级一致,但在所有端到端对比中均无序列匹配(0/8, 0/16, 0/64)。关键发现是整数免责:在已验证无溢出条件下,INT32点积精确且顺序无关,累加器不可能是差异来源。对Qwen3-1.7B(196层)和8B(252层)各线性层输入相同操作数,2的幂次缩放下输出比特一致(196/196, 252/252),真实缩放下差异不超过1个bfloat16间距。分歧被精准定位至缩放应用与输出舍入阶段。作为探测检查点,同一干预使端到端比特一致恢复(8/8, 16/16)。跨实现的FP8 GEMM则显示差异随缩减深度增加,而INT8差异保持在百万分之一量级且在64倍K范围内≤1间距。教师强制回放将层与词元关联:翻转集中在小逻辑值边际,预测翻转风险的ROC-AUC达0.94(16,384位置)。我们将发布预注册结果、逐层预测、内核选择证据清单及一致性检测规程。
原文摘要 · Abstract (English)
Two GPU kernels implementing the same scaled INT8 GEMM interface are usually treated as interchangeable. We test that assumption: holding the checkpoint, prompts, hardware, inference engine, decoding, and quantization configuration fixed, we swap only the INT8 linear kernel (CUTLASS versus Triton) inside vLLM. At 1.7B each arm reproduces itself bit-for-bit across cold restarts, yet the arms agree on no sequence in any end-to-end comparison we ran (0/8, 0/16, and 0/64). What makes this more than a benchmark discrepancy is an integer alibi: for shared INT8 operands under a verified no-overflow bound, the INT32 dot product is exact and order-independent, so the accumulator cannot be the source of any difference. Feeding both kernels identical operands from every linear layer of Qwen3-1.7B and 8B (196 and 252 layers), we find bit-identical outputs under power-of-two scales, confirming a pinned prediction list 196/196 and 252/252 (pre-registered at 1.7B, pinned but not blind at 8B), and observed differences of at most one bfloat16 spacing under the checkpoints' real scales. This localizes the divergence to scale application and output rounding after the exact accumulator. Applied as a probe checkpoint, the same intervention restores end-to-end bitwise agreement (8/8 and 16/16 sequences). Cross-implementation FP8 GEMM shows a different signature: both the prevalence and the magnitude of differences grow with reduction depth, while the INT8 fraction stays at parts per million and within one spacing over a 64x range of K. Teacher-forced replay ties layers to tokens: flips concentrate at small logit margins, which predict flip risk with ROC-AUC 0.94 on 16,384 positions. We will release the pre-registration, per-layer predictions, manifests with kernel-selection evidence, and a conformance procedure that turns these controls into a concrete check for kernel interchangeability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。