用数据压缩技术将大模型权重压到0.5比特，大幅降低显存占用。
UltraSketchLLM: Sub-1-Bit LLM Compression via Sketch and Hardware-Friendly Operators
- 通过数据摘要技术实现权重压缩，突破1比特限制
- 达到0.5比特/权重的压缩率，显存峰值降低
- 硬件友好设计，推理速度提升14.9倍，适合边缘部署
大型语言模型（LLMs）当前需要大量GPU显存，亟需高效且极端的权重压缩方法。现有方法或受制于每权重1比特的理论极限，或导致性能严重下降与效率低下。为在资源受限场景部署LLM，我们提出UltraSketchLLM，利用数据摘要技术压缩模型权重，将峰值显存占用显著降低，压缩率可达0.5比特/权重。结合硬件友好实现，UltraSketchLLM保持可接受的性能损失，且相比朴素摘要方案实现14.9倍加速，延迟开销极低。
原文摘要 · Abstract (English)
Large language models (LLMs) require larger GPU memory size these days, necessitating efficient and extreme weight compression methods. Existing compression methods are either theoretically limited by 1 bit per weight or face severe performance degradation and inefficiency. To deploy LLMs in resource-constrained scenarios, we introduce UltraSketchLLM, compressing LLMs with data sketch. It reduces peak GPU memory footprint with a high compression rate down to 0.5 bit per weight. Combined with hardware-friendly implementation, UltraSketchLLM keeps tolerable performance degradation and extremely low latency overhead with 14.9x speedup compared to naive sketch solution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。