用缓存机制加速大模型推理，无需训练且兼容现有系统
Cacheback: Speculative Decoding With Nothing But Cache
- 仅用LRU缓存表记录词元局部性生成推测序列
- 性能达同类方法顶尖水平，延迟降低30%以上
- 适合希望快速部署推理加速的工程团队
我们提出Cacheback推理，一种无需训练、与模型无关的推测解码方法，利用语言中的局部性特征加速大语言模型（LLM）推理。Cacheback仅依赖词元n-gram的最近最少使用（LRU）缓存表生成候选序列，设计极简却性能卓越，在同类方法中表现领先。其简单架构便于集成到现有系统中，同时具备快速适配新领域的能力。
原文摘要 · Abstract (English)
We present Cacheback Decoding, a training-free and model-agnostic speculative decoding method that exploits the locality in language to accelerate Large Language Model (LLM) inference. Cacheback leverages only Least Recently Used (LRU) cache tables of token n-grams to generate draft sequences. Cacheback achieves state-of-the-art performance among comparable methods despite its minimalist design, and its simplicity allows easy integration into existing systems. Cacheback also shows potential for fast adaptation to new domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。