构建大规模细粒度视频推理数据集，提升视觉语言模型理解能力
VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection
- 通过语义感知选帧减少冗余，结合GPT-4o生成高质量问答对
- 引入视频思维链标注，实现多模态证据驱动的逻辑推理
- 适合研究视频理解、多模态推理与大模型微调的学者使用
大型视觉语言模型（LVLMs）在多模态理解上取得显著进展，但视频推理任务仍受限于高质量、大规模数据集的缺乏。现有视频问答（VideoQA）数据集或依赖昂贵的人工标注且粒度不足，或采用自动构建方法导致逐帧分析冗余，制约了复杂推理的可扩展性与有效性。为此，我们提出VideoEspresso，一个包含保留关键空间细节与时间连贯性的视频问答对，并附带中间推理步骤的多模态标注。其构建流程采用语义感知方法降低冗余，再利用GPT-4o生成问答对。进一步开发视频思维链（CoT）标注，引导GPT-4o从问答对和视频内容中提取逻辑关系。为充分利用高质量视频问答对，我们提出混合式LVLM协作框架，包含帧选择器与两阶段指令微调的推理型LVLM，可自适应选择核心帧并基于多模态证据进行思维链推理。在包含14个任务的基准上评估，相较9种主流LVLM，该方法在多数任务上表现更优，验证了其卓越的视频推理能力。代码与数据集将开源：https://github.com/hshjerry/VideoEspresso
原文摘要 · Abstract (English)
The advancement of Large Vision Language Models (LVLMs) has significantly improved multimodal understanding, yet challenges remain in video reasoning tasks due to the scarcity of high-quality, large-scale datasets. Existing video question-answering (VideoQA) datasets often rely on costly manual annotations with insufficient granularity or automatic construction methods with redundant frame-by-frame analysis, limiting their scalability and effectiveness for complex reasoning. To address these challenges, we introduce VideoEspresso, a novel dataset that features VideoQA pairs preserving essential spatial details and temporal coherence, along with multimodal annotations of intermediate reasoning steps. Our construction pipeline employs a semantic-aware method to reduce redundancy, followed by generating QA pairs using GPT-4o. We further develop video Chain-of-Thought (CoT) annotations to enrich reasoning processes, guiding GPT-4o in extracting logical relationships from QA pairs and video content. To exploit the potential of high-quality VideoQA pairs, we propose a Hybrid LVLMs Collaboration framework, featuring a Frame Selector and a two-stage instruction fine-tuned reasoning LVLM. This framework adaptively selects core frames and performs CoT reasoning using multimodal evidence. Evaluated on our proposed benchmark with 14 tasks against 9 popular LVLMs, our method outperforms existing baselines on most tasks, demonstrating superior video reasoning capabilities. Our code and dataset will be released at: https://github.com/hshjerry/VideoEspresso
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。