通过分层异构缓存，显著降低长文本生成的内存与延迟开销。
Shallow Prefill, Deep Decoding: Efficient Long-Context Inference via Layer-Asymmetric KV Visibility
- 仅在底层缓存提示词键值对，解码阶段保持全层可见
- 128K上下文下减少25%活跃KV内存，首字响应快33%
- 适合需要高效长文本生成的应用场景
在仅解码器语言模型中，长提示词在预填充阶段需全层缓存并反复访问，导致成本高昂。本文提出浅预填充、深解码（SPEED）策略，仅在低层保留提示词键值对，解码阶段则保持全层可见。实验表明，在仅使用75%层进行预填充时，Llama-3.1-8B模型在OLMES类基准上平均得分51.2（全层基线为51.4），首字时间（TTFT）提升33%，每输出一个词时间（TPOT）改善22%，128K上下文时活跃KV内存减少25.0%。层级诊断显示该策略保留了关键提示选择与表征稳定区域。结果表明，解码阶段全层可见时，提示词无需全程全层缓存。
原文摘要 · Abstract (English)
Long-context inference in decoder-only language models is costly because long prompts are processed during Prefill, cached at every layer, and repeatedly attended to during autoregressive Decode. We introduce \emph{Shallow Prefill, dEEp Decode} (SPEED), a phase-asymmetric KV-visibility policy that materializes non-anchor prompt-token KV states only in lower layers while keeping Decode-phase tokens full-depth. Unlike previous approaches that make upper-layer prompt KV states cheaper to store or construct, SPEED removes prefill tokens from the upper-layer Decode visibility set altogether. With a minimal BoS anchor, this simple change preserves broad benchmark quality while reducing long-context cost. In a controlled Llama-3.1-8B instruction-tuning study, SPEED using only 75\% of layers for prefill tokens reaches 51.2 average score on OLMES-style benchmarks, compared with 51.4 for the full-depth baseline, while improving TTFT by 33\%, TPOT by 22\%, and reducing active KV memory by 25.0\% at 128K context. Layer-wise diagnostics suggest that this cutoff retains the main prompt-selection and representation-stabilization regions of the full-depth model. These results show that long-context prompt tokens need not always persist as full-depth KV-cache objects when Decode-phase tokens remain full-depth.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。