arXiv:2412.09613cs.CV2024-12CVPR被引 29

统一图像视频压缩,用渐进编码保留细节,不损失图像性能。

PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models

  • 将图像转为静态视频,逐帧渐进压缩补全信息。
  • 每帧仅64个令牌,仍能保持时空细节,视频任务领先。
  • 兼容图像视频统一处理,适合多模态应用开发。

大型视觉语言模型已扩展至理解图像与视频。视觉令牌压缩被用于降低视觉输入的长序列长度。为满足不同任务需求,现有高性能模型通常对图像和视频分别采用不同的压缩策略,限制了图像与视频的联合处理能力。为此,我们将每张图像扩展为“静态视频”,提出一种统一的渐进视觉令牌压缩方法(PVC):逐帧逐步编码并自适应压缩,以补充前帧未提取的信息。利用视频固有的时序冗余高效压缩视频令牌;图像以静态视频形式重复,空间细节可在多帧中逐步补充。PVC实现了图像与视频的统一令牌压缩。在每帧仅64个令牌(默认)的情况下,仍可有效保留空间细节与时间变化。实验表明,该模型在多个视频理解基准上达到当前最优性能,涵盖长视频与细粒度短视频任务。同时,其统一压缩策略在图像基准上无性能损失,尤其在细节敏感任务中表现优异。

原文摘要 · Abstract (English)

Large Vision-Language Models (VLMs) have been extended to understand both images and videos. Visual token compression is leveraged to reduce the considerable token length of visual inputs. To meet the needs of different tasks, existing high-performance models usually process images and videos separately with different token compression strategies, limiting the capabilities of combining images and videos. To this end, we extend each image into a "static" video and introduce a unified token compression strategy called Progressive Visual Token Compression (PVC), where the tokens of each frame are progressively encoded and adaptively compressed to supplement the information not extracted from previous frames. Video tokens are efficiently compressed with exploiting the inherent temporal redundancy. Images are repeated as static videos, and the spatial details can be gradually supplemented in multiple frames. PVC unifies the token compressing of images and videos. With a limited number of tokens per frame (64 tokens by default), spatial details and temporal changes can still be preserved. Experiments show that our model achieves state-of-the-art performance across various video understanding benchmarks, including long video tasks and fine-grained short video tasks. Meanwhile, our unified token compression strategy incurs no performance loss on image benchmarks, particularly in detail-sensitive tasks.

视觉压缩多模态模型视频生成统一处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。