优化小模型部署：量化让Whisper-small更小更快
Quantizing Whisper-small: How design choices affect ASR performance
- 对比四种工具库，系统评估量化方案对性能影响
- 动态int8量化使模型缩小57%，错误率还更低
- 适合想在设备端高效部署语音模型的人
大型语音识别模型如Whisper-small虽准确度高，但计算开销大，难以部署于边缘设备。为此，我们对Whisper-small开展统一的跨库后训练量化（PTQ）评估，分离分析量化方案、方法、粒度与位宽的影响。实验基于PyTorch、Optimum-Quanto、HQQ和bitsandbytes四类库，在LibriSpeech test-clean和test-other上进行。结果表明，使用Quanto的动态int8量化表现最佳，模型尺寸减少57%，且词错误率优于基线；静态量化表现较差，可能源于Whisper的Transformer结构；更激进的格式（如nf4、int3）可实现最高71%压缩，但在嘈杂环境下损失精度。整体证明，合理选择PTQ方法可在无需微调的情况下显著降低模型大小与推理成本，实现Whisper-small在资源受限硬件上的高效部署。
原文摘要 · Abstract (English)
Large speech recognition models like Whisper-small achieve high accuracy but are difficult to deploy on edge devices due to their high computational demand. To this end, we present a unified, cross-library evaluation of post-training quantization (PTQ) on Whisper-small that disentangles the impact of quantization scheme, method, granularity, and bit-width. Our study is based on four libraries: PyTorch, Optimum-Quanto, HQQ, and bitsandbytes. Experiments on LibriSpeech test-clean and test-other show that dynamic int8 quantization with Quanto offers the best trade-off, reducing model size by 57% while improving on the baseline's word error rate. Static quantization performed worse, likely due to Whisper's Transformer architecture, while more aggressive formats (e.g., nf4, int3) achieved up to 71% compression at the cost of accuracy in noisy conditions. Overall, our results demonstrate that carefully chosen PTQ methods can substantially reduce model size and inference cost without retraining, enabling efficient deployment of Whisper-small on constrained hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。