通过压缩键值缓存实现同态加速，显著降低长文本推理延迟
HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
- 直接在量化后的键值数据上计算，跳过耗时的反量化步骤
- 实测作业完成时间减少最高达70.9%，优于现有方法52.3%
- 适合大规模分布式推理场景，尤其长序列输入任务
分离式大语言模型推理通过将计算密集的prefill阶段与内存密集的decode阶段解耦，避免了两者干扰并提升资源利用率。然而，两阶段间键值（KV）数据传输成为瓶颈，尤其在长提示场景下更为严重。此外，prefill与decode的计算开销直接影响作业完成时间（JCT），KV数据规模对长序列而言可能达到不可承受程度。现有KV量化方法虽能缓解传输压力并减少内存占用，但引入显著的反量化开销，进一步增加计算延迟。本文提出同态加速键值缓存压缩方法（HACK），消除昂贵的反量化步骤，直接在量化后的KV数据上进行计算，以近似方式降低高成本的矩阵乘法开销。基于实际工作负载轨迹的大量实验表明，HACK相较分离式推理基线可降低70.9%的作业完成时间，相比最先进的KV量化方法也降低了52.3%。
原文摘要 · Abstract (English)
Disaggregated Large Language Model (LLM) inference has gained popularity as it separates the computation-intensive prefill stage from the memory-intensive decode stage, avoiding the prefill-decode interference and improving resource utilization. However, transmitting Key-Value (KV) data between the two stages can be a bottleneck, especially for long prompts. Additionally, the computation time overhead for prefill and decode is key for optimizing Job Completion Time (JCT), and KV data size can become prohibitive for long prompts and sequences. Existing KV quantization methods can alleviate the transmission bottleneck and reduce memory requirements, but they introduce significant dequantization overhead, exacerbating the computation time. We propose Homomorphic Acceleration via Compression of the KV cache (HACK) for disaggregated LLM inference. HACK eliminates the heavy KV dequantization step, and directly performs computations on quantized KV data to approximate and reduce the cost of the expensive matrix-multiplication step. Extensive trace-driven experiments show that HACK reduces JCT by up to 70.9% compared to disaggregated LLM inference baseline and by up to 52.3% compared to state-of-the-art KV quantization methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。