Minima通过结构化压缩让大模型在低显存下高效运行,提升推理速度。
A Practical Tensor-Network Compression Pipeline for Production-Scale Large Language Models
- 用轻量卷积预测层和块的敏感度,选择性压缩低敏感区域。
- 压缩后显存从64GiB降至40GiB,单请求吞吐提升至75 tokens/s。
- 适合部署大模型的生产环境,尤其支持高并发与推测解码。
大语言模型因GPU显存和推理延迟受限,难以大规模部署。我们提出Minima,一个面向生产的压缩流水线,可学习Transformer中哪些部分适合结构性压缩,并转化为实际服务收益。Minima训练轻量卷积预测器以估计层级与块级敏感度,对低敏感区域采用Tucker、张量-列车和张量-环分解的混合策略,进行短时修复微调,并使用自定义Triton与CUDA核执行操作。压缩后显存减少,使小规模草稿模型与大验证模型的推测解码成为可能。在Qwen3-32B、8k上下文窗口下,峰值显存由64 GiB降至40 GiB;单请求吞吐从基准的40提升至50(Minima)和75(推测解码),50个并发请求下分别达到34、44、53 tokens/s,表明在高并发下仍具有效性。我们对比了近期张量网络、低秩+量化及跨层共享方法,认为Minima是实现更激进结构压缩的实用路径,基于共享张量主干与极小逐层适配器。
原文摘要 · Abstract (English)
Large language models are limited in deployment by GPU memory and inference latency. We present Minima, a production compression pipeline that learns where and how to structurally compress a Transformer and turns that compression into real serving gains. Minima trains a lightweight convolutional predictor to estimate layer- and patch-level sensitivity, applies a mixture of Tucker, tensor-train, and tensor-ring decompositions to low-sensitivity regions, performs a short healing fine-tune, and executes the resulting operators with custom Triton and CUDA kernels. The reduced memory footprint enables speculative decoding with a small draft model and a larger verifier. On Qwen3-32B at an 8k-token context window, Minima reduces peak VRAM from 64 GiB to 40 GiB. For a single active request, throughput increases from 40 tokens per second (baseline) to 50 tokens per second (Minima) and 75 tokens per second (Minima with speculative decoding). Under 50 parallel requests, throughput is 34, 44, and 53 tokens per second respectively, showing that Minima remains effective under high concurrency even when speculative decoding gains compress. We position Minima relative to recent tensor-network, low-rank plus quantization, and cross-layer sharing methods, and argue that it is a practical step toward more aggressive structural compression via shared tensor backbones with tiny per-layer adapters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。