让图像模型轻松看懂视频,通过动态选帧和拆解问题提升理解能力。
D-CoDe: Scaling Image-Pretrained VLMs to Video via Dynamic Compression and Question Decomposition
- 动态选帧+内容聚合,减少冗余保留关键视觉信息。
- 将复杂问题拆解为子问题,缓解模型输入过载。
- 无需训练即可提升多任务视频理解,尤其擅长长视频分析。
视频大语言模型(Vid-LLMs)可通过适配图像预训练的视觉语言模型(VLMs)构建,但面临密集且时序延展的视觉输入超出模型处理能力的挑战。本文指出感知瓶颈与令牌过载是核心障碍。为此提出D-CoDe——一种无需训练的适配框架,结合动态压缩与问题分解:动态压缩通过自适应选择代表性帧并进行内容感知的空间令牌聚合,降低冗余同时保留关键信息;问题分解则将原查询重构为子问题,引导模型聚焦视频不同方面,实现更全面理解。实验表明,D-CoDe显著提升多种基准上的视频理解性能,尤其在长视频基准上表现突出,展现了处理复杂视频-语言任务的潜力。代码已开源。
原文摘要 · Abstract (English)
Video large language models (Vid-LLMs), which excel in diverse video-language tasks, can be effectively constructed by adapting image-pretrained vision-language models (VLMs). However, this adaptation remains challenging, as it requires processing dense and temporally extended visual inputs that exceed the capacity of image-based models. This paper identifies the perception bottleneck and token overload as key challenges in extending image-based VLMs to the video domain. To address these issues, we propose D-CoDe, a training-free adaptation framework that incorporates dynamic compression and question decomposition. Specifically, dynamic compression alleviates the perception bottleneck through adaptive selection of representative frames and content-aware aggregation of spatial tokens, thereby reducing redundancy while preserving informative content. In parallel, question decomposition mitigates token overload by reformulating the original query into sub-questions, guiding the model to focus on distinct aspects of the video and enabling more comprehensive understanding. Experiments demonstrate that D-CoDe effectively improves video understanding across various benchmarks. Furthermore, strong performance on the challenging long-video benchmark highlights the potential of D-CoDe in handling complex video-language tasks. Code is available at https://github.com/hukcc/D-CoDe.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。