通过语义相似性复用中间激活，实现Transformer推理加速。
LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
- 按层缓存中间结果，基于输入语义相似性匹配
- 在BERT和GPT-2上实现最高3.1倍加速，准确率损失<0.5%
- 适用于编码器与解码器架构，支持任意层缓存
基于Transformer的语言模型在众多任务中表现出色，但其高推理延迟制约了实时与大规模部署。现有缓存机制如词元级键值缓存虽能提升自回归解码速度，但应用范围有限。本文提出LLMCache，一种新的分层缓存框架，通过基于输入序列语义相似性的中间激活复用，加速Transformer推理。该方法无需依赖具体模型，可应用于编码器与解码器架构，并支持任意层缓存。我们引入轻量级指纹匹配机制以识别语义相似输入，并设计自适应淘汰策略缓解缓存过时问题。在SQuAD、WikiText-103和OpenBookQA上的实验表明，相较于基线模型，推理时间最多提速3.1倍，准确率下降小于0.5%。结果表明，LLMCache是一种实用且通用的Transformer推理优化方案。
原文摘要 · Abstract (English)
Transformer-based language models have achieved remarkable performance across a wide range of tasks, yet their high inference latency poses a significant challenge for real-timeand large-scale deployment. While existing caching mechanisms,such as token-level key-value caches, offer speedups in autore-gressive decoding, they are limited in scope and applicability. In this paper, we present LLMCache, a novel layer-wise caching framework that accelerates transformer inference by reusing intermediate activations based on semantic similarity of input sequences. Unlike prior work, LLMCache is model-agnostic,operates across both encoder and decoder architectures, and supports caching at arbitrary transformer layers. We introduce a lightweight fingerprinting mechanism for matching seman-tically similar inputs and propose adaptive eviction strategies to manage cache staleness. Experiments on BERT and GPT-2 across SQuAD, WikiText-103, and OpenBookQA show up to 3.1 X speedup in inference time with <0.5% accuracy degradation. Our results highlight LLMCache as a practical and general-purpose solution for optimizing transformer inference in real-world applications
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。