arXiv:2505.12434cs.CV2025-05NeurIPS被引 94

通过强化微调提升多模态大模型的视频推理能力。

VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuning

  • 构建认知启发式提示管道,生成高质量视频思维链数据。
  • 在6个基准上达到当前最佳性能,显著提升推理准确率。
  • 适合研究视频理解、多模态推理与强化学习的学者使用。

强化微调(RFT)在提升大语言模型(LLM)的人类级推理能力方面展现出巨大潜力,并已拓展至多模态大模型(MLLMs)。然而,由于视频数据中固有的复杂逻辑、时序和因果结构,视频推理仍是持续挑战。为此,我们提出VideoRFT,一种将RFT范式扩展至培养MLLM类人视频推理能力的新方法。VideoRFT遵循标准两阶段流程:基于思维链(CoT)标注的监督微调(SFT),随后通过强化学习(RL)提升泛化能力。核心挑战在于缺乏大规模高质量视频CoT数据集,我们通过多专家驱动、认知启发式CoT构建管道解决此问题。首先,设计认知启发式提示策略,仅依赖视频内容的丰富、结构化、字面表示,由推理型LLM生成初步CoT;随后,由条件于实际视频的MLLM对CoT进行修订,确保视觉一致性并减少视觉幻觉。该流程生成两个新数据集:用于SFT的VideoRFT-CoT-102K 和用于RL的VideoRFT-RL-310K。为增强RL阶段,引入新型语义一致性奖励,显式促进文本推理与视觉证据的一致性,鼓励模型生成基于视觉输入、连贯且情境感知的推理输出。大量实验表明,VideoRFT在六个视频推理基准上均取得当前最优性能。

原文摘要 · Abstract (English)

Reinforcement fine-tuning (RFT) has shown great promise in achieving humanlevel reasoning capabilities of Large Language Models (LLMs), and has recently been extended to MLLMs. Nevertheless, reasoning about videos, which is a fundamental aspect of human intelligence, remains a persistent challenge due to the complex logic, temporal and causal structures inherent in video data. To fill this gap, we propose VideoRFT, a novel approach that extends the RFT paradigm to cultivate human-like video reasoning capabilities in MLLMs. VideoRFT follows the standard two-stage scheme in RFT: supervised fine-tuning (SFT) with chain-of-thought (CoT) annotations, followed by reinforcement learning (RL) to improve generalization. A central challenge to achieve this in the video domain lies in the scarcity of large-scale, high-quality video CoT datasets. We address this by building a multi-expert-driven, cognition-inspired CoT curation pipeline. First, we devise a cognition-inspired prompting strategy to elicit a reasoning LLM to generate preliminary CoTs based solely on rich, structured, and literal representations of video content. Subsequently, these CoTs are revised by a MLLM conditioned on the actual video, ensuring visual consistency and reducing visual hallucinations. This pipeline results in two new datasets, i.e.VideoRFT-CoT-102K for SFT and VideoRFT-RL-310K for RL. To further strengthen the RL phase, we introduce a novel semantic-consistency reward that explicitly promotes the alignment between textual reasoning and visual evidence. This reward encourages the model to produce coherent, context-aware reasoning outputs grounded in visual input. Extensive experiments show that VideoRFT achieves state-of-the-art performance on six video reasoning benchmarks.

视频推理强化微调多模态思维链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。