arXiv:2510.07486cs.CL2025-10被引 2

提出异步稀疏解码框架,让大模型推理更快更省显存。

AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decoding

  • 用短期查询预测当前查询状态,无需等待解码循环。
  • 异步解耦缓存过滤与生成过程,实现计算重叠,吞吐提升超20%。
  • 适合高并发长推理任务,显著降低延迟,适用于生产级部署。

测试时扩展(TTS)通过长链式思维(CoT)提升大模型推理能力,但线性增长的键值缓存加剧了显存瓶颈。现有查询感知的页级稀疏解码虽在有限算力下表现优异,却受限于依赖顺序的页筛选和粗粒度标记选择,导致高并发与长CoT场景下服务效率低,甚至运行时间超过前向推理本身。本文首次发现,当前步骤的查询状态可从近期短窗口查询中统一准确近似,从而实现无需训练的查询感知稀疏化。我们提出AsyncSpade:一个基于两个核心组件的异步高效TTS框架——(1) 轻量级时序回归模块,用于预测下一标记查询状态;(2) 异步解耦架构,将KV缓存过滤从自回归解码环中分离,通过异步机制使标记级缓存选择与前向计算并行。据我们所知,这是首个在不牺牲性能前提下消除序列依赖的方法。在配备A100节点的典型部署环境中验证,AsyncSpade完全重叠了KV缓存操作与推理流水线,达到理论最优每输出标记时间(TPOT)。具体而言,相比最先进基线Quest,TPOT降低超20%;相比全注意力机制,在Qwen3-8B与Qwen3-32B模型上均实现至少50%的TPOT下降,同时在AIME-24/25、GPQA-Diamond、MATH-500等多个TTS基准上保持或超越其精度。

原文摘要 · Abstract (English)

Test-time scaling (TTS) boosts LLM reasoning via long chain-of-thought (CoT), but the linear KV-cache growth amplifies the memory-bound bottleneck of LLM decoding. Query-aware page-level sparse decoding can achieve state-of-the-art performance under constrained FLOPs budgets, but is limited by both sequential-dependent page filtering and coarse-grained token selection, hampering serving efficiency and model performance on TTS tasks under high concurrency and long CoT scenarios (consuming even higher runtime than the forward pipeline itself). In this paper, we first find that the current-step query state can be accurately approximated in a unified manner from a short window of recent queries, enabling training-free query-aware sparsity without waiting in the decoding loop. We propose AsyncSpade, an asynchronous framework for efficient TTS built on two core components: (1) a novel light-weight temporal-regressive module that predicts the next-token query state; (2) an asynchronous and disaggregated framework that decouples the KV cache filtering from the auto-regressive decoding loop, overlapping the token-level KV selection with the forward inference computation through asynchronism. To our knowledge, AsyncSpade is the first to eliminate the sequential dependence without sacrificing model performance. We validate the effectiveness of AsyncSpade on common LLM serving setups with an A100 node, where AsyncSpade fully overlaps KV-cache operations with the inference pipeline, achieving theoretical optimal time-per-output-token (TPOT). Specifically, AsyncSpade delivers over 20% reduction on TPOT compared to SoTA baseline (i.e. Quest) and at least 50% TPOT reduction compared to full attention on Qwen3-8B and Qwen3-32B models, while matching or surpassing their accuracy on various TTS benchmarks (AIME-24/25, GPQA-Diamond, MATH-500).

大模型推理稀疏解码异步架构显存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。