arXiv:2603.18489cs.CL2026-03被引 2

用解码词熵判断何时重算,显著加速扩散语言模型推理。

EntropyCache: Decoded Token Entropy Guided KV Caching for Diffusion Language Models

  • 基于解码词分布熵判断缓存是否过时,决策成本恒定。
  • 在标准和链式思考任务上提速15.2至26.4倍,精度几乎不变。
  • 无需训练,适合高并发、长上下文场景的高效推理部署。

基于扩散的大型语言模型(dLLMs)依赖双向注意力,无法实现无损键值(KV)缓存,导致每个去噪步骤需完整前向传播。现有近似KV缓存方法通过选择性更新缓存状态降低开销,但其决策代价随上下文长度或模型深度增加。我们提出EntropyCache,一种无需训练的KV缓存方法,利用新解码词分布的最大熵作为恒定成本信号,决定是否重算。该设计基于两个经验观察:(1)解码词熵与KV缓存漂移相关,可作为缓存过时的廉价代理;(2)解码词特征波动在解掩后持续多步,建议重算最近的$ k $个解码词。跳过或重算的决策每步仅需$ O(V) $计算,与上下文长度和模型规模无关。在LLaDA-8B-Instruct和Dream-7B-Instruct上的实验表明,EntropyCache在标准基准上实现15.2×–26.4×加速,在链式思考基准上实现22.4×–24.1×加速,精度保持良好,决策开销仅占推理时间的0.5%。代码已开源。

原文摘要 · Abstract (English)

Diffusion-based large language models (dLLMs) rely on bidirectional attention, which prevents lossless KV caching and requires a full forward pass at every denoising step. Existing approximate KV caching methods reduce this cost by selectively updating cached states, but their decision overhead scales with context length or model depth. We propose EntropyCache, a training-free KV caching method that uses the maximum entropy of newly decoded token distributions as a constant-cost signal for deciding when to recompute. Our design is grounded in two empirical observations: (1) decoded token entropy correlates with KV cache drift, providing a cheap proxy for cache staleness, and (2) feature volatility of decoded tokens persists for multiple steps after unmasking, motivating recomputation of the $k$ most recently decoded tokens. The skip-or-recompute decision requires only $O(V)$ computation per step, independent of context length and model scale. Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct show that EntropyCache achieves $15.2\times$-$26.4\times$ speedup on standard benchmarks and $22.4\times$-$24.1\times$ on chain-of-thought benchmarks, with competitive accuracy and decision overhead accounting for only $0.5\%$ of inference time. Code is available at https://github.com/mscheong01/EntropyCache.

扩散模型推理加速缓存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。