用多帧拼接成面板，提升长视频理解模型性能。
Video Panels for Long Video Understanding
- 将多帧图像拼成面板，以提升时间分辨率
- 在长视频数据集上最高提升19.4%准确率
- 无需训练、可适配现有任意视频模型
近期视频-语言模型（VLMs）在长视频理解任务中表现良好，但仍逊于图像或短视频任务。为提升长上下文建模能力，研究者常引入新模块或增加复杂度。本文另辟蹊径：不依赖有限数据微调模型，而是最大化现有模型性能。提出一种专为长视频设计的视觉提示策略——将多个帧组合成面板，以空间细节换取时间分辨率。该方法无需训练、无额外参数、与模型无关，可无缝集成至现有VLM。在五个涵盖多种模型架构、规模和上下文窗口的基准上验证，结果一致有效。在最长视频数据集TimeScope (Long)上，视频问答准确率最高提升19.4%。本方法显著提升了长视频理解水平。代码已公开：https://fedespu.github.io/Video-Panels。
原文摘要 · Abstract (English)
Recent Video-Language Models (VLMs) achieve promising results on long-video understanding, but their performance still lags behind that achieved on tasks involving images or short videos. This has led to great interest in improving the long context modeling of VLMs by introducing novel modules and additional complexity. In this paper, we take a different approach: rather than fine-tuning VLMs with the limited data available, we attempt to maximize the performance of existing models. To this end, we propose a novel visual prompting strategy specifically designed for long-video understanding. By combining multiple frames as panels into one image, we effectively trade off spatial details for temporal resolution. Our approach is training-free, parameter-free, and model-agnostic, and can be seamlessly integrated into existing VLMs. Extensive experiments on five established benchmarks across a wide range of model architectures, sizes, and context windows confirm the consistency of our approach. For the TimeScope (Long) dataset, which has the longest videos, the accuracy for video question answering is improved by up to 19.4%. Overall, our method raises the bar for long video understanding models. The code is available at https://fedespu.github.io/Video-Panels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。