arXiv:2508.06038cs.CVcs.AI2025-08被引 2

用频域压缩视觉令牌,大幅提升多模态模型效率

Fourier Compressor: Frequency-Domain Visual Token Compression for Vision-Language Models

  • 在频域通过FFT去除视觉表示冗余,无需额外参数
  • 压缩后保留96%以上原精度,推理计算量减少83.8%
  • 适用于图像与视频任务,跨模型通用性强

视觉-语言模型因高分辨率图像和视频输入产生大量视觉令牌,导致显著计算开销和推理延迟。现有无参数令牌压缩方法多依赖令牌选择或合并,易丢失视觉信息或扭曲表示分布,高压缩比下性能明显下降。为此,我们探索频域压缩策略:基于频率变换在图像压缩(如JPEG)中的成功经验,系统分析视觉表示的频率冗余,发现语义信息在不同频带中分布不均。据此提出Fourier Compressor,一种高效、无参数、高度泛化的频域令牌压缩模块。该模块通过FFT实现,复杂度为$/mathcal{O}(n^2 /log n)$,无额外参数,计算开销极低但能保持语义保真。在图像基准上实验表明,该方法在保留超过96%原始精度的同时,推理浮点运算量最多降低83.8%,生成速度提升31.2%,优于现有无参数方法,甚至超越部分有参方法。其在LLaVA和Qwen-VL架构上表现一致,且可拓展至视频理解任务,展现出良好的实用性。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) incur substantial computational overhead and inference latency due to the large number of vision tokens introduced by high-resolution image and video inputs. Existing parameter-free token compression methods typically rely on token selection or merging, yet they risk discarding substantial visual information or distorting the original representation distribution, resulting in pronounced performance degradation at high compression ratios. In response, we aim to explore a more effective and efficient visual token compression strategy, with a promising direction in the frequency domain. Motivated by the success of frequency-domain transforms in image compression (e.g., JPEG), we systematically analyze the frequency redundancy in visual representations and uncover a non-uniform distribution of semantic information across frequency bands. Building upon this, we introduce Fourier Compressor, an effective, parameter-free, and highly generalizable module that removes redundancy from visual representations within the frequency domain. Implemented via FFT with $\mathcal{O}(n^2 \log n)$ complexity and no additional parameters, Fourier Compressor introduces negligible computational overhead while preserving semantic fidelity. Extensive experiments on image-based benchmarks demonstrate that our method achieves a favorable performance-efficiency trade-off, retaining over 96% of the original accuracy while reducing inference FLOPs by up to 83.8% and boosting generation speed by 31.2%. It consistently outperforms existing parameter-free methods and even surpasses some parameterized approaches. Importantly, Fourier Compressor generalizes consistently across both LLaVA and Qwen-VL architectures, and further extends to video understanding tasks, highlighting its practical applicability for efficient VLMs.

视觉压缩频域处理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。