通过分步推理任务提升视频模型的细粒度理解能力。
CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks
- 将复杂视频问题拆解为定位、追踪、时空关系提取四类基础任务。
- 在NeXT-QA上，模型平均得分提升3.3至17.4点，因果推理提升14.6点。
- 适合需要精细视频推理的科研与工业应用，如智能监控、自动驾驶。
尽管视频大语言模型（VideoLLMs）取得进展，但如何让模型具备基于细粒度物体级视频理解的链式思维（CoT）推理能力仍是关键挑战。现有指令微调模型（如Qwen和LLaVA系列）依赖高层视频-文本对训练，缺乏支持组合性、分步推理的结构化标注。本文提出CoTasks：一种链式思维驱动的视频指令微调框架，将现有数据集（如NeXT-QA、STAR）中的复杂视频问题分解为四类实体级基础任务：帧定位、实体追踪、空间与时间关系提取。通过在输入中嵌入这些中间推理步骤，CoTasks使模型能够显式进行以对象为中心的时空推理。在NeXT-QA基准上的实验表明，该方法显著提升推理性能：LLaVA-video-7B在GPT-4评估中平均得分提升+3.3点，Qwen2.5-VL-3B提升+17.4点，其中因果推理(+14.6)、时间推理(+10.9)和描述性推理(+48.1)子类别提升尤为明显。结果证明CoTasks作为结构化链式思维监督框架，在提升组合式视频推理方面具有有效性。
原文摘要 · Abstract (English)
Despite recent progress in video large language models (VideoLLMs), a key open challenge remains: how to equip models with chain-of-thought (CoT) reasoning abilities grounded in fine-grained object-level video understanding. Existing instruction-tuned models, such as the Qwen and LLaVA series, are trained on high-level video-text pairs, often lacking structured annotations necessary for compositional, step-by-step reasoning. We propose CoTasks: Chain-of-Thought based Video Instruction Tuning Tasks, a new framework that decomposes complex video questions of existing datasets (e.g., NeXT-QA, STAR) into four entity-level foundational tasks: frame localization, entity tracking, spatial and temporal relation extraction. By embedding these intermediate CoT-style reasoning steps into the input, CoTasks enables models to explicitly perform object-centric spatiotemporal reasoning. Experiments on the NeXT-QA benchmark show that CoTasks significantly enhance inference performance: LLaVA-video-7B improves by +3.3 points in average GPT-4 evaluation score, and Qwen2.5-VL-3B gains +17.4, with large boosts in causal (+14.6), temporal (+10.9), and descriptive (+48.1) subcategories. These results demonstrate the effectiveness of CoTasks as a structured CoT-style supervision framework for improving compositional video reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。