arXiv:2506.00318cs.CV2025-06中稿 · the IEEE/CVF Confe…被引 32

让视频大模型像思考一样逐帧推理,提升理解准确性。

Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoning

  • 单阶段推理直接关联关键帧,避免时间错位
  • 在多个视频理解任务上准确率显著提升
  • 合成数据也能有效提升真实场景表现

近期研究表明,在回答用户问题前,让大语言模型生成自然语言推理链可显著提升性能。该方法已扩展至多模态大模型,使其对图像和视频内容生成思维链(CoT)。针对视频输入,现有工作采用复杂多步流程提取并包含相关帧,或仅生成简单单阶段推理,导致时间定位不准。本文提出首个支持单阶段推理的视频大模型,能明确引用关键帧,减少推理过程中的时间不一致。方法简洁统一,无需额外模块进行帧选择或描述生成。我们构建了大规模的COF-DATA数据集,包含自然与合成视频的多样化问题、答案及对应的帧级推理链。通过在该数据集上微调视频大模型,其生成的推理链能准确识别回答问题所需的关键帧,从而在多个视频理解基准测试中持续提升性能。令人意外的是,仅使用合成数据进行训练,模型在真实世界基准上的准确率也显著提高。代码已开源。

原文摘要 · Abstract (English)

Recent work has shown that eliciting Large Language Models (LLMs) to generate reasoning traces in natural language before answering the user's request can significantly improve their performance across tasks. This approach has been extended to multimodal LLMs, where the models can produce chains-of-thoughts (CoT) about the content of input images and videos. For video inputs, prior works use complex multi-step pipelines that extract and include relevant frames from videos in the CoT, or produce simpler single-stage reasoning traces at the expense of poor temporal grounding. Here, we propose the first video LLMs with single-stage reasoning that includes explicit references to relevant frames, thereby reducing temporal inconsistencies in the reasoning process. Our approach is simple, unified, and self-contained, employing a single-stage inference to handle complex video understanding tasks without relying on auxiliary modules for frame selection or caption generation. For this, we first create COF-DATA, a large dataset of diverse questions, answers, and corresponding frame-grounded reasoning traces from both natural and synthetic videos, spanning various topics and tasks. Our models, obtained fine-tuning video LLMs on this chain-of-frames (CoF) data, generate reasoning traces that accurately identify key frames to answer given questions. In turn, this consistently improves performance across multiple video understanding benchmarks. Surprisingly, we find that synthetic data alone, despite being out-of-distribution with respect to these real-world benchmarks, provides a significant boost in model accuracy. Code is available at https://github.com/SaraGhazanfari/CoF.

视频理解多模态思维链合成数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。