70B大模型在单张A40上实现33GB压缩，推理速度达57词元/秒
Budget-Aware Compression Pipeline for Single-GPU LLM Inference: Methods, Trade-offs, and Coupling Effects
- 基于内存、吞吐、成本三重预算，设计压缩流水线
- 层间剪枝提升量化鲁棒性，稀疏缓存补强INT8量化
- 提供可复现评估协议，支持自动化管道搜索
在NVIDIA GPU上部署700亿参数语言模型面临设备内存、长序列吞吐和工程集成成本的限制。本文将单卡推理视为三维度预算约束下的设计问题，研究剪枝、量化与键值缓存压缩在真实执行中的交互关系。控制性消融实验表明，层间剪枝使权重量化更鲁棒；键值缓存稀疏化在不降低解码速度的前提下减少内存占用，与INT8键值量化互补，而静态向量量化常与动态缓存机制冲突。基于上述耦合效应和显式预算追踪，构建了实用压缩流水线：将70B模型压缩至约33GB，单张A40上处理10,000词元提示时维持约57词元/秒的速度，且在通用及推理基准上绝对精度保持在5%以内。论文贡献了设计规则与可复现评估协议，联合报告质量、内存与端到端速度，并为真实单卡约束下的自动化流水线搜索奠定基础。
原文摘要 · Abstract (English)
Single-GPU deployment of 70B-parameter language models on an NVIDIA GPU is constrained by device memory, long-context throughput, and engineering integration cost. We cast single-GPU inference as a budget-aware design problem over these three axes and study how pruning, quantization, and KV-cache compression interact under realistic execution. Controlled ablations show that layer-wise pruning makes weight quantization more robust. KV-cache sparsification complements INT8 KV quantization by reducing memory without hurting decoding speed, while static vector quantizers often conflict with dynamic caching. Guided by these coupling results and explicit budget tracking, we assembled a practical pipeline and compressed a 70B model to about 33 GB, sustained about 57 tokens/s on 10k token prompts on a single A40, and kept absolute accuracy within 5% on common and reasoning benchmarks. We contribute design rules and a reproducible evaluation protocol that jointly report quality, memory, and end-to-end speed, and we provide a foundation for automated pipeline search under realistic single-GPU constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。