用自监督强化学习让视频理解模型更准更稳地思考。
Reinforcing Structured Chain-of-Thought for Video Understanding
- 用总结-思考-作答的结构化思维流程，替代依赖标注的复杂训练。
- 在7个视频问答数据集上达到当前最优性能。
- 自动调节推理多样性，避免思维僵化，适合需要可靠推理的场景。
多模态大语言模型在视频理解方面展现潜力，但其推理常出现思维漂移和时间理解弱的问题，即使采用组相对策略优化（GRPO）等强化学习方法也难以解决。现有强化学习方法通常依赖监督微调（SFT），需昂贵的思维链（CoT）标注和多阶段训练，且固定推理路径限制泛化能力，可能引入偏差。为此，我们提出摘要驱动的强化学习（SDRL），一种无需SFT的单阶段强化学习框架，采用结构化思维链格式：总结 -> 思考 -> 回答。SDRL在GRPO目标中引入两项自监督机制：1）视觉知识一致性（CVK）通过降低生成摘要间的KL散度，强化事实依据；2）动态推理多样性（DVR）根据群体准确率动态调节思考多样性，促进探索。该设计有效平衡对齐与探索，同时监督最终答案与推理过程。实验表明，该方法在7个公开视频问答数据集上达到当前最优表现。
原文摘要 · Abstract (English)
Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize -> Think -> Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。