用小波变换分离视频时空信息,实现10倍压缩仍保持99.6%性能
WaveZip: Wavelet-Driven Space-Time Decoupling for Video Token Condensation

- 通过小波变换解耦视频时空特征,分别处理时间冗余与空间显著性
- 在10倍压缩比下仍保持99.6%原始性能,优于现有方法
- 无需训练可直接接入主流视觉语言模型,适合长视频理解场景
现有大型视觉语言模型(LVLMs)在长视频理解中受限于视觉令牌的二次计算开销。尽管近期高效方法尝试通过硬剪枝或均匀合并压缩令牌,但均局限于空间特征域,导致结构上下文与语义细节纠缠。本文提出WaveZip,一种联合信号-频率域的高效视频推理框架。基于时序冗余存在于低通近似尺度、空间显著性与高频成分强相关的洞察,利用离散小波变换(DWT)分离信号。时间上,采用1维DWT分析查询帧相关性,高频系数结合帧间差异进行门控,共同驱动精确的帧级令牌预算分配;空间上,2维DWT将特征分解为低频近似与高频细节,高频系数在查询显著区域调制以控制空间重建。重要的是,WaveZip无需任务特定训练,可无缝集成至现成的LVLMs中提升推理效率。大量实验表明,在长视频理解基准上,其在极端10倍压缩比下仍保留99.6%的完整性能,持续优于当前最优方法。
原文摘要 · Abstract (English)
Existing Large Vision-Language Models (LVLMs) struggle with long-form video understanding due to the quadratic computational cost of visual tokens. While recent efficient methods attempt to compress tokens via hard pruning or uniform merging, they operate strictly in the spatial feature domain, where robust structural context and discriminative semantic details are inherently entangled. In this work, we propose WaveZip, a joint signal-frequency-domain framework for efficient video inference. Driven by the insight that temporal redundancy resides in low-pass approximation scales while spatial saliency strongly correlates with high-frequency components, WaveZip leverages Discrete Wavelet Transforms (DWT) to disentangle these signals. Temporally, it employs 1D DWT to analyze query-frame relevance, and the resulting high-frequency coefficients are further gated by inter-frame differences, with both signals jointly driving the dynamic allocation of a precise frame-level token budget. Spatially, a 2D DWT decomposes features into low-frequency approximations and high-frequency detail components, where the high-frequency coefficients are modulated within query-salient regions to regulate spatial reconstruction. Importantly, WaveZip requires no task-specific training and can be seamlessly integrated into off-the-shelf LVLMs to boost inference efficiency. Extensive experiments on long video understanding benchmarks demonstrate that WaveZip retains 99.6% of the full performance under an extreme 10x compression ratio, consistently outperforming state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。