优化低秩Transformer推理速度，实现2.55倍解码加速
FlashSVD v1.5: Making Low-Rank Transformers Inference Actually Fast
- 统一低秩压缩模型的运行时表示，减少执行碎片化
- 通过专用内核与图重放技术，实现最高2.55倍解码加速
- 适合关注大模型推理优化的研究者与工程师
基于SVD的低秩压缩虽能减少Transformer参数和理论计算量，但实际服务速度提升有限。我们发现这主要是运行时问题：因子化检查点导致执行路径碎片化，且预填充与自回归解码的开销差异显著。本文提出FlashSVD v1.5，一种统一的低秩Transformer推理运行时。该框架将多种公开的SVD压缩方法映射到统一的因子化表示，并结合分阶段专用内核、密集键值缓存解码、打包MLP执行及每层CUDA图重放，重构了低秩推理路径，使其成为轻量级运行时。在典型解码服务场景下，FlashSVD v1.5实现了最高2.55倍的解码加速和2.39倍的端到端加速；在多个主流SVD压缩方法上，平均解码加速达1.48倍，平均端到端加速1.44倍。结果表明，实用的低秩加速需要运行时协同设计，而不仅是压缩算法改进。代码已开源：https://github.com/Zishan-Shao/FlashSVD。
原文摘要 · Abstract (English)
SVD-based Low-rank compression reduces transformer parameters and nominal FLOPs, but these savings often translate poorly into real LLM serving speedups. We show that this gap is largely a runtime problem: factorized checkpoints fragment execution paths, and the resulting overhead differs substantially between prefill and autoregressive decode. We present FlashSVD v1.5, a unified inference runtime for serving SVD-compressed transformers. FlashSVD v1.5 maps diverse public SVD compression families to a common factorized representation and combines phase-specific kernels with dense-KV decode, packed MLP execution, and per-layer CUDA-graph replay to reorganize the low-rank serving path into a thin runtime. Across representative decoder-serving settings, FlashSVD v1.5 achieves up to 2.55x decode and 2.39x end-to-end speedup, and it attains 1.48x average decode and 1.44x average end-to-end speedup across multiple popular SVD compression families. These results suggest that practical low-rank acceleration requires runtime co-design, not compression algorithms alone. Our code is available at: https://github.com/Zishan-Shao/FlashSVD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。