用时空图自动生成带推理链的训练数据,提升视频大模型的复杂推理能力。
STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training
- 基于时空场景图引导,从原始视频自动生成带思维链的问答数据。
- 在三步以上推理任务中提升21.3%,仅用少量自生成数据即达优异表现。
- 适合需要强时序推理的视频理解研究者和开发者使用。
视频大语言模型在基础视频理解任务(如字幕生成、粗粒度问答)上表现良好,但在需跨对象关系、交互与事件进行多步时空推理的组合推理任务中仍存在困难。现有挑战包括人工标注成本高、数据缺乏时空组合性、以及缺少显式推理监督。本文提出STEP方法,一种基于时空图的自训练框架,使视频大模型能从任意原始视频中自动生成富含推理过程的微调数据以自我提升。首先构建多样视频的时空场景图(STSG),捕捉细粒度、多粒度的视频语义;随后利用STSG引导生成包含思维链(CoT)推理路径的多步问答数据,将答案与推理过程共同作为训练目标,以增强模型对显式推理步骤的学习能力。实验表明,该方法在不同规模模型上均有效,三步及以上推理任务性能提升21.3%。同时,仅需极少量自生成的带推理链样本,即可在组合推理与综合理解基准上取得领先表现,验证其广泛适用性与巨大潜力。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) have recently shown strong performance in basic video understanding tasks, such as captioning and coarse-grained question answering, but struggle with compositional reasoning that requires multi-step spatio-temporal inference across object relations, interactions, and events. The hurdles to enhancing this capability include extensive manual labor, the lack of spatio-temporal compositionality in existing data and the absence of explicit reasoning supervision. In this paper, we propose STEP, a novel graph-guided self-training method that enables Video-LLMs to generate reasoning-rich fine-tuning data from any raw videos to improve itself. Specifically, we first induce Spatio-Temporal Scene Graph (STSG) representation of diverse videos to capture fine-grained, multi-granular video semantics. Then, the STSGs guide the derivation of multi-step reasoning Question-Answer (QA) data with Chain-of-Thought (CoT) rationales. Both answers and rationales are integrated as training objective, aiming to enhance model's reasoning abilities by supervision over explicit reasoning steps. Experimental results demonstrate the effectiveness of STEP across models of varying scales, with a significant 21.3\% improvement in tasks requiring three or more reasoning steps. Furthermore, it achieves superior performance with a minimal amount of self-generated rationale-enriched training samples in both compositional reasoning and comprehensive understanding benchmarks, highlighting the broad applicability and vast potential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。