arXiv:2411.02327cs.CV2024-11中稿 · ICLR被引 21

通过提示引导压缩视频帧,实现高效长视频理解。

PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance

  • 根据用户指令动态筛选关键视觉区域,减少冗余信息。
  • 支持18倍视觉标记压缩,仍保持领先性能。
  • 适合需要处理长视频的多模态推理场景。

过去一年,基于视频的大语言模型在处理长视频方面取得显著进展,尤其依赖极长上下文长度。然而,大量视觉标记导致计算开销急剧增加,效率成为主要瓶颈。本文发现根本原因在于视频内容的高度冗余。为此,我们提出一种新型池化策略,可在大幅压缩视觉标记的同时保留与指令相关的关键语义。提出的PPLLaVA模型包含三个核心组件:基于CLIP的视觉-提示对齐模块,用于根据用户指令识别感兴趣区域;提示引导的自适应池化机制,采用类似卷积的池化方式压缩视觉序列;以及针对复杂长提示对话设计的片段上下文扩展模块。在高达18倍的标记压缩率下,PPLLaVA在多种视频理解基准上保持优异表现,涵盖从图像到视频的任务(如字幕生成、问答)及长视频推理任务,并显著提升推理吞吐量。代码已开源:https://github.com/farewellthree/PPLLaVA。

原文摘要 · Abstract (English)

In the past year, video-based large language models (Video LLMs) have achieved impressive progress, particularly in their ability to process long videos through extremely extended context lengths. However, this comes at the cost of significantly increased computational overhead due to the massive number of visual tokens, making efficiency a major bottleneck. In this paper, we identify the root of this inefficiency as the high redundancy in video content. To address this, we propose a novel pooling strategy that enables aggressive token compression while retaining instruction-relevant visual semantics. Our model, Prompt-guided Pooling LLaVA (PPLLaVA), introduces three key components: a CLIP-based visual-prompt alignment module that identifies regions of interest based on user instructions, a prompt-guided pooling mechanism that adaptively compresses the visual sequence using convolution-style pooling, and a clip context extension module tailored for processing long and complex prompts in visual dialogues. With up to 18x token reduction, PPLLaVA maintains strong performance across tasks, achieving state-of-the-art results on diverse video understanding benchmarks-ranging from image-to-video tasks such as captioning and QA to long-form video reasoning-while significantly improving inference throughput. Codes have been available at https://github.com/farewellthree/PPLLaVA.

视频理解提示引导效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。