用进度条辅助视频时间定位，让大模型更懂视频时序推理
VTimeCoT: Thinking by Drawing for Video Temporal Grounding and Reasoning
- 引入进度条和高亮工具，实现无需训练的视频时序理解增强
- 在Qwen2VL-7B和GPT4o上显著提升视频定位与推理准确率
- 支持可解释的跨模态思维链，适合需要透明决策的视频分析场景
近年来，基于多模态大语言模型（MLLM）的视频问答受到广泛关注，得益于大语言模型的显著进展。然而，这些模型在视频时间定位与推理任务中存在明显不足，制约了真实世界视频理解系统的发展。受人类使用视频播放器进度条进行视频理解的启发，我们提出VTimeCoT——一种无需训练的高效框架，用于提升视频定位与推理性能。该框架引入两种新颖的视觉工具：即插即用的进度条集成模块与高效高亮工具。此外，为克服传统文本思维链（CoT）的局限，我们设计了跨模态的时空思维链（visuotemporal CoT），实现视频与文本间的联合推理。实验表明，该方法在Qwen2VL-7B和GPT4o基线上均显著提升了视频时间定位与推理问答的表现。最终，框架实现了可组合且可解释的推理过程。
原文摘要 · Abstract (English)
In recent years, video question answering based on multimodal large language models (MLLM) has garnered considerable attention, due to the benefits from the substantial advancements in LLMs. However, these models have a notable deficiency in the domains of video temporal grounding and reasoning, posing challenges to the development of effective real-world video understanding systems. Inspired by how humans use video players to interact with the progress bar for video comprehension, we introduce VTimeCoT, a simple yet effective training-free framework, designed for high-performance video grounding and reasoning. The proposed framework incorporates two novel visual tools of the progress bar: a plug-and-play progress bar integration tool and a high-efficiency highlighting tool. In addition, to address the limitations of conventional text-based chain-of-thought (CoT) approaches, we introduce a visuotemporal CoT process that integrates cross-modality reasoning across both video and text. Our approach demonstrates significant performance improvements on both Qwen2VL-7B and GPT4o baselines in tasks of video temporal grounding and reasoning-based question answering. Finally, we showcase that the proposed framework achieves a compositional and interpretable reasoning process. Project page: https://vtimecot.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。