用轻量语言模型动态压缩视频令牌，复杂场景多描述，简单场景少描述。
Seeing More, Saying More: Lightweight Language Experts are Dynamic Video Token Compressors
- 用语言模型生成软标题作为视觉表征，实现动态压缩。
- 相比VideoGPT+降低49%计算量，性能仍保持领先。
- 适合追求高效视频理解的开发者与研究者使用。
大型视频-语言模型在视频理解任务中取得显著进展，但其效率受限于海量视觉令牌的处理。现有令牌压缩方法采用固定压缩比，忽略了不同视频片段语义密度的差异，导致信息丰富的片段因令牌不足而表达不充分，静态或内容贫乏片段则产生冗余计算。为此，我们提出LangDC：一种语言感知的动态令牌压缩器。LangDC利用轻量语言模型对视频片段进行描述，将视频转换为软标题令牌作为视觉表征，并通过语义密度感知监督训练，旨在同时满足两个目标：1）保留下游任务推理所需的关键视觉线索；2）根据描述长度动态调整压缩比例，反映场景丰富程度。该设计模仿人类认知机制：复杂场景（看得更多）会生成更详细的语言描述（说得更多），简单场景则用简短描述。实验表明，相比VideoGPT+，本方法在保持竞争力性能的同时降低49%的浮点运算量。定性结果也显示，方法能根据视频片段内容丰富度自适应调整压缩比例。
原文摘要 · Abstract (English)
Recent advancements in large video-language models have revolutionized video understanding tasks. However, their efficiency is significantly constrained by processing high volumes of visual tokens. Existing token compression strategies apply a fixed compression ratio, ignoring the variability in semantic density among different video clips. Consequently, this lead to inadequate representation of information-rich clips due to insufficient tokens and unnecessary computation on static or content-poor ones. To address this, we propose LangDC, a Language-aware Dynamic Token Compressor. LangDC leverages a lightweight language model to describe video clips, converting them into soft caption tokens as visual representations. Trained with our proposed semantic density-aware supervision, LangDC aims to 1) cover key visual cues necessary for downstream task reasoning and 2) dynamically adjust compression ratios based on scene richness, reflected by descriptions length. Our design mimics how humans dynamically express what they see: complex scenes (seeing more) elicit more detailed language to convey nuances (saying more), whereas simpler scenes are described with fewer words. Experimental results show that our method reduces FLOPs by 49% compared to VideoGPT+ while maintaining competitive performance. Furthermore, qualitative results demonstrate our approach adaptively adjusts the token compression ratio based on video segment richness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。