视频微调让模型学会帧间推理，提升长视频问答能力。
Video Finetuning Improves Reasoning Between Frames
- 设计显式帧间推理流程，生成连续帧的事件过渡描述。
- 视频微调模型在长视频问答中表现显著优于图像模型。
- 微调后模型可将时间推理能力迁移到静态图像任务中。
多模态大语言模型在视觉理解方面进展迅速，但将其从图像扩展到视频时，常采用简单的帧标记拼接方式。本文研究视频微调对多模态大模型的影响，提出视觉思维链（vCoT），通过生成连续帧间的过渡事件描述，实现显式帧间推理。我们系统比较了仅图像训练的模型与视频微调后的模型，无论是否具备过渡提示。实验表明，vCoT能显著提升图像模型在长视频问答上的表现，而对已微调的视频模型提升有限，说明后者已隐式学习帧间关系。此外，视频模型在纯静态关系推理任务中也优于图像模型基线，展现出时间推理能力的跨模态迁移。
原文摘要 · Abstract (English)
Multimodal large language models (LLMs) have made rapid progress in visual understanding, yet their extension from images to videos often reduces to a naive concatenation of frame tokens. In this work, we investigate what video finetuning brings to multimodal LLMs. We propose Visual Chain-of-Thought (vCoT), an explicit reasoning process that generates transitional event descriptions between consecutive frames. Using vCoT, we systematically compare image-only LVLMs with their video-finetuned counterparts, both with and without access to these transitional cues. Our experiments show that vCoT significantly improves the performance of image-only models on long-form video question answering, while yielding only marginal gains for video-finetuned models. This suggests that the latter already capture frame-to-frame transitions implicitly. Moreover, we find that video models transfer this temporal reasoning ability to purely static settings, outperforming image models' baselines on relational visual reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。