提出WnW机制，用20%显存实现长音频语音大模型近全缓存精度。
WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
- 按离线校准将注意力头分为锚点、潮汐和固定三类，动态管理缓存。
- 在长音频上保持接近全缓存的精度，仅需20%的显存存储音频片段。
- 适合长语音生成任务，尤其对显存受限场景有显著优势。
长时音频输入使语音大模型中的键值（KV）缓存成为主要内存开销。现有仅预填充阶段压缩的方案一旦丢弃音频位置便无法恢复，导致在长音频上表现脆弱：预填充注意力集中在音频起始区域（注意力汇聚效应），而解码时注意力分布广泛，两者重叠度低。本文提出WnW（Waxing-and-Waning KV Cache），通过离线校准将KV头分类为锚点、潮汐和固定三类。锚点头保留在GPU中，提供解码时各标记读取来源的信号；潮汐头在CPU驻留补充内容，根据聚合锚点头得分逐块召回；固定头仅保留部分在GPU，其余永久丢弃。在LibriSpeech-Long数据集上，使用两个3B规模模型（Voxtral-mini-3b 和 Qwen2.5-Omni-3B），WnW在仅保留20%音频片段于GPU的情况下仍保持近全缓存精度，而预填充基线方法无法完成推理。结果在语言、任务和领域迁移下均具泛化性，且实测中CPU-GPU召回带来的解码延迟极小。
原文摘要 · Abstract (English)
Long-form audio inputs make the KV cache the dominant memory cost of speech LLMs. Prefill-only KV compression methods permanently discard audio KV positions once evicted, with no pathway to recover them during decoding. We show this is fragile on long-form audio: prefill attention concentrates near the audio start (an attention-sink effect), while decode-time attention distributes broadly, and the two rankings overlap weakly. We propose WnW (Waxing-and-Waning KV cache), which classifies KV-heads into anchor, tidal, and fixed roles via offline calibration. Anchor heads keep all audio KV on GPU and yield a decode-time signal of which audio region each token is read from; tidal heads keep a CPU-resident complement that is recalled chunk-by-chunk based on aggregated anchor-head scores; fixed heads keep only an on-GPU subset, with the rest permanently discarded. On LibriSpeech-Long with two 3B backbones (Voxtral-mini-3b and Qwen2.5-Omni-3B), WnW preserves near-Full-Cache accuracy while keeping only 20% of audio tokens on GPU, where prefill-only baselines fail to terminate. Results generalize across language, task, and domain shifts, and CPU-GPU recall adds little decode-time overhead in our measurements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。