用小波变换筛选关键信息，让扩散语言模型处理长文本更准更快。
WaveFilter: Enhancing the Long-Context Capability of Diffusion LLMs via Wavelet-Guided KV Cache Filtering
- 通过小波变换分解长序列，精准识别重要词元
- 构建稀疏键值缓存，使长文本生成质量提升显著
- 无需训练、即插即用，适配主流缓存方法
扩散大语言模型（DLMs）在多项任务中表现出显著优势。然而，受限于多步迭代推理机制，其在长上下文任务中的计算开销和推理延迟已成为制约大规模部署的核心瓶颈。处理长序列时，现有键值（KV）缓存机制常面临生成质量急剧下降的困境，核心挑战在于如何在超长上下文中精确高效地过滤关键词元。受人类阅读过程启发，我们提出 extbf{WaveFilter}，一种通用且无需训练的缓存框架。该框架创新性地引入小波变换对长序列进行分解，以实现关键词元的精准识别，并据此构建稀疏键值缓存，用于计算最终的上下文表征。实验表明，WaveFilter作为即插即用的通用框架，显著提升了现有主流KV缓存方法在复杂长上下文任务中的性能。
原文摘要 · Abstract (English)
Diffusion Large Language Models (DLMs) have demonstrated significant advantages across various tasks. However, constrained by their multi-step iterative inference mechanism, their computational overhead and inference latency in long-context tasks have become core bottlenecks restricting their large-scale deployment. When processing long sequences, existing Key-Value (KV) caching mechanisms often face a dilemma where generation quality degrades drastically, where the core challenge lies in precisely and efficiently filtering critical tokens within ultra-long contexts. Inspired by the human reading process, we propose \textbf{WaveFilter}, a universal and training-free caching framework. This framework innovatively introduces the wavelet transform for decomposition of long sequences to achieve precise identification of key tokens, based on which a sparse KV Cache is constructed to compute the final contextual representation. Experimental results demonstrate that WaveFilter, as a plug-and-play generic framework, significantly enhances the performance of existing mainstream KV Cache methods in complex long-context tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。