在服务器CPU上实现小模型高效INT8推理，无需额外工具链
Efficient INT8 Inference of Small NLP Models on Server CPUs with PyTorch Native Stack
- 将SmoothQuant集成到PyTorch原生栈，结合图级融合与优化内核
- 在BERT等模型上实现最高5.8倍吞吐提升，精度损失可忽略
- 适合需要低延迟、高吞吐的工业级小模型部署场景
小型NLP模型（尤其是BERT族编码器）在分类、排序和检索等工业负载中仍具重要地位。在服务器CPU上，INT8量化能带来理想的延迟-吞吐-成本权衡，但用户越来越期望这种加速能直接通过原生PyTorch实现。本文将SmoothQuant集成至TorchAO，利用TorchInductor进行图级融合，并在oneDNN、AVX512_VNNI和AMX三种实现间智能选择高效INT8 GEMM内核。在BERT、DistilBERT和XLM-RoBERTa基准测试中，该方法相比FP32基线实现最高5.8倍端到端吞吐提升，精度损失极小甚至不可测。通过屋顶模型进行详细性能分析验证了有效性。实现已合并至PyTorch和TorchAO，支持原生PyTorch工具链开箱即用。
原文摘要 · Abstract (English)
Small NLP models, especially BERT-family encoders, remain important in industrial workloads such as classification, ranking, and retrieval even in the era of large language models. On server CPUs, INT8 quantization offers an attractive latency-throughput-cost trade-off, but users increasingly expect such acceleration to be available directly in the native PyTorch stack. We integrate SmoothQuant into TorchAO and optimize the resulting inference path for Intel Xeon CPUs through graph-level fusion in TorchInductor and efficient INT8 GEMM kernel selection across oneDNN-, AVX512_VNNI-, and AMX-based implementations. Across BERT, DistilBERT, and XLM-RoBERTa benchmarks, the approach delivers up to 5.8x end-to-end throughput speedup with negligible---and in some cases no measurable---accuracy loss relative to the FP32 baseline. We also validated our work by detailed performance analysis with roofline models. The implementation has been upstreamed to PyTorch and TorchAO, enabling out-of-the-box deployment with native PyTorch tooling
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。