用语言提示动态删减视频帧，提速65%还保持高精度
Language-Guided Temporal Token Pruning for Efficient VideoLLM Processing
- 根据问题中的时间线索智能筛选视频帧，保留关键段落
- 计算量减少65%，在多个数据集上性能仅下降1%-3%
- 适合需要高效处理长视频的场景，尤其对带时间标记的问题更优
视觉语言模型（VLMs）在处理长视频时受限于注意力机制的二次复杂度。本文提出语言引导的时间令牌剪枝（LGTTP），利用查询中的时间线索自适应地剪枝视频令牌，在保持上下文连贯性的同时降低计算开销。与均匀剪枝或关键帧选择不同，LGTTP在时间相关片段中保留更高的令牌密度。该模型无关框架可集成至TimeChat和LLaVA-Video，实现65%的计算量减少，同时保持97%-99%的原始性能。在QVHighlights上，HIT@1提升9.5%；在Charades-STA上，R@1保留99.6%。其在含明确时间标记的查询中表现尤为出色，并在通用视频理解任务中保持有效性。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) struggle with long-form videos due to the quadratic complexity of attention mechanisms. We propose Language-Guided Temporal Token Pruning (LGTTP), which leverages temporal cues from queries to adaptively prune video tokens, preserving contextual continuity while reducing computational overhead. Unlike uniform pruning or keyframe selection, LGTTP retains higher token density in temporally relevant segments. Our model-agnostic framework integrates with TimeChat and LLaVA-Video, achieving a 65% reduction in computation while preserving 97-99% of the original performance. On QVHighlights, LGTTP improves HIT@1 by +9.5%, and on Charades-STA, it retains 99.6% of R@1. It excels on queries with explicit temporal markers and remains effective across general video understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。