arXiv:2608.20961cs.AIcs.CL2026-08

提出TreeWY方法,解决混合模型推测解码的内存瓶颈问题。

TreeWY: Speculative Verification for Gated DeltaNet Hybrids

  • 用树形WY变换重构建模门控增量规则,避免为每个草稿节点存储状态快照。
  • 在相同接受率下,将推测解码的递归状态内存和KV缓存压力降低,释放显存提升吞吐与首字延迟。
  • 适用于高宽草稿树场景,尤其适合显存受限的推理服务部署。

现代开源模型多为混合结构:多数层采用线性注意力的门控增量网络(Gated DeltaNet, GDN)层,携带固定大小的循环状态而非不断增长的键值(KV)缓存。这使常规解码内存高效,但损害推测解码效率。当前系统需在每个草稿位置对GDN层快照完整循环状态,且这些快照无法跨草稿树分支共享,导致宽而高接受率的草稿树因内存不足不可行。本文提出无需快照的方法:利用门控增量规则的树状WY变换,仅通过一次三角求解即可计算每个草稿节点输出,并在确认时重构唯一接受的状态,仅存储一个小的伪值矩阵而非各节点状态;推导仅依赖门控增量规则,不依赖其他架构细节。在两个规模的同一混合模型族(Qwen3.5 35B 和 397B)上进行服务基准测试,该方法在相同接受长度下显著降低推测解码的递归状态内存与KV缓存压力,释放的HBM可转化为更高吞吐与更低首字延迟,当内存受限时效果明显,而在内存充裕时仅付出少量开销。对草稿树宽度而言,同等内存可支持更宽、更高接受率的草稿生成,虽未立即带来吞吐提升,但实现可行性突破。

原文摘要 · Abstract (English)

Modern open models are hybrids: most layers are linear-attention (Gated DeltaNet, GDN) layers carrying a small fixed-size recurrent state instead of a growing key-value (KV) cache. This makes ordinary decoding memory-efficient, but hurts speculative decoding. To verify a batch of draft tokens and then roll back the rejected ones, today's systems snapshot the full recurrent state at every draft position for GDN layers, and those snapshots cannot be shared across branches of a draft tree, so a wide, high-acceptance tree becomes memory-infeasible. We remove the snapshots. Using a tree-structured WY transform of the gated delta rule, we compute every draft node's output with a single triangular solve and reconstruct only the one accepted state on commit, storing a small pseudo-value matrix instead of per-node states; the derivation depends only on the gated delta rule, not on any other architectural detail. In serving benchmarks on two scales of one hybrid model family (Qwen3.5 35B and 397B) this cuts speculative recurrent-state memory and KV-cache pressure at identical acceptance length, turning the freed HBM into higher throughput and much lower time-to-first-token (TTFT) wherever memory binds, and costing a few percent where it does not. For tree width the same memory buys affordability: a wider, higher-acceptance draft becomes possible, though not yet a throughput win.

推测解码门控增量内存优化大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。