用量化+推测解码，让40亿参数大模型在低端显卡上快7倍
Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
- 目标模型量化后用感知蒸馏恢复精度，保持原始量化网格
- 训练专用块扩散起草模型，两阶段适配低精度目标
- 通过量化与滑动窗口注意力，显著降低长文本解码延迟
本文介绍我们在高效Qwen竞赛中的方案，目标是在资源受限的NVIDIA A10G GPU上实现低延迟服务。系统结合量化目标模型与推测解码：为恢复精度，对目标模型应用量化感知蒸馏并保留原始量化网格；为加速解码，使用两阶段训练法训练专用于量化目标模型的块扩散起草模型，先从高精度目标学习，再适配低精度目标。由于起草模型每步都被调用，我们进一步通过量化和滑动窗口注意力减少其开销，在保持草稿词接受率的同时提升长上下文解码延迟表现。最终提交结果相较基线平均提速6.978倍，满足质量要求，位列第三。代码与资源已开源。
原文摘要 · Abstract (English)
This report describes our approach to the Efficient Qwen Competition, where the goal is to enable low-latency serving of Qwen3.5-4B on a resource-constrained NVIDIA A10G GPU. Our system combines a quantized target model with speculative decoding. To recover accuracy, we apply quantization-aware distillation to the target model while retaining the original quantization grid. To speed up decoding, a block-diffusion drafter specialized for the quantized target model is trained using a two-stage procedure: first learning from the high-precision target and then adapting to the low-precision target. Because the drafter is invoked at every speculative decoding step, we further reduce its overhead with quantization and sliding-window attention, preserving draft-token acceptance while improving long-context decoding latency. As a result, our submission achieves a 6.978$\times$ average speedup over the baseline while satisfying the required quality thresholds, ranking 3rd overall. We hope these results provide useful insights for practical LLM inference. The code and resources are available at https://github.com/nota-github/adaptfm-quant-dflash
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。