针对低比特大模型推理中的误差不均问题,提出自适应补偿系统提升性能。
SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving

- 按输入动态调整纠错强度,避免误修正简单词元
- 在关键层部署轻量纠错模块,恢复W4与FP16间56%-75%的困惑度差距
- 兼顾低延迟与可预测性,适合生产环境高效部署
高效的大语言模型服务日益受部署成本制约。量化是降低服务成本的关键技术,但即使最先进的4比特量化器在小模型上仍存在显著质量损失。我们发现根本原因在于量化误差高度依赖输入,不同词元间差异大,而现有后量化补偿方法为静态固定修正,导致简单词元被过度纠正,困难词元仍欠纠正。为此,提出SPEAR系统,通过基于每词元门控的轻量级误差补偿器(ECs),仅在CKA引导的熵感知诊断识别出的最敏感层部署,使有限参数预算聚焦有效位置。为解决补偿器部署带来的系统挑战——额外计算、输入依赖门控引发的张量并行同步及配置间延迟波动——SPEAR采用自适应内核融合调度,结合尾部集成对等归约内核与点对点双写,将后补偿计算融入低比特GEMM;同时设计满足服务等级协议(SLO)的EC感知调度器,实现可预测的服务性能。在具有挑战性的逐通道量化设置下,SPEAR在不足1%模型内存开销下,恢复了W4与FP16之间56%-75%的困惑度差距,并保持与主流4比特部署相当的延迟。
原文摘要 · Abstract (English)
Efficient large language model (LLM) serving is increasingly constrained by deployment cost. Quantization is a key technique for reducing serving cost, yet even state-of-the-art 4-bit quantizers exhibit a noticeable quality gap from FP16, particularly for smaller models where low-bit serving is most beneficial. We identify a fundamental cause of this gap: quantization error is highly input-dependent and varies substantially across tokens, while existing post-quantization compensation methods are static and apply identical corrections to all inputs. As a result, easy tokens are over-corrected while hard tokens remain under-corrected. We present SPEAR, a system for post-quantization error-adaptive recovery that improves low-bit LLM serving. SPEAR introduces lightweight Error Compensators (ECs) modulated by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic. This focuses a small parameter budget where it is most effective. Efficient deployment of ECs presents several systems challenges, including additional computation, tensor-parallel synchronization caused by input-dependent gating, and latency instability across configurations. SPEAR addresses these issues through adaptive kernel-fusion dispatch, combining an epilogue-integrated peer-reduction kernel with P2P dual-write to fuse the post-EC computation into low-bit GEMMs, and an SLO-constrained EC-aware scheduler for predictable serving performance. Across challenging per-channel quantization settings, SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latency comparable to a widely used 4-bit serving deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。