不改模型结构，用四招提速大模型推理
QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
- 动态停顿、缓存跳过、上下文融合、自适应量化四法协同
- 最多节省39.6%计算量，困惑度下降不超过0.2
- 适合部署在已有模型上，无需重训练或改架构
推理占大语言模型部署中超过90%的延迟和能耗。尽管训练效率已大幅提升，但运行时优化仍是瓶颈，尤其在自回归解码下。现有方法如剪枝、量化、提前退出和推测解码，常需重训练或修改架构，破坏解码兼容性。本文提出QuickSilver，一个模块化、逐标记级的推理优化框架，在不改变模型权重与结构的前提下实现语义自适应。该框架集成四项协同机制：(i) 动态标记停顿，对表征收敛的标记停止计算；(ii) KV缓存跳过，选择性跳过内存写入以降低注意力开销；(iii) 上下文标记融合，将冗余标记合并为共享路径以缩短序列长度。不同于推测解码或MoE路由，QuickSilver仅作用于冻结的密集模型，无需辅助网络。在GPT-2和Llama-2上应用于WikiText-103与C4数据集，最高可减少39.6%的浮点运算量，困惑度下降不超过0.2。
原文摘要 · Abstract (English)
Inference accounts for the majority of latency and energy consumption in large language model (LLM) deployments, often exceeding 90% of total cost. While training-time efficiency has seen extensive progress, runtime optimization remains a key bottleneck, particularly under autoregressive decoding. Existing approaches -- such as pruning, quantization, early exits, and speculative decoding -- often require retraining, architectural changes, or disrupt decoding compatibility. We introduce QuickSilver, a modular, token-level framework that enables semantic adaptivity at inference time without altering model weights or structure. QuickSilver integrates four synergistic mechanisms: (i) Dynamic Token Halting, which halts computation for tokens with converged representations; (ii) KV Cache Skipping, which selectively suppresses memory writes to reduce attention overhead; and (iii) Contextual Token Fusion, which collapses redundant tokens into shared paths to shrink sequence length. Unlike speculative decoding or MoE routing, QuickSilver operates entirely on frozen, dense models and requires no auxiliary networks. Applied to GPT-2 and Llama-2 across WikiText-103 and C4, QuickSilver achieves up to 39.6% FLOP reduction with negligible perplexity degradation (<=0.2).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。