通过树状线索推理提升视频理解,减少幻觉并增强细粒度感知。
Video-ToC: Video Tree-of-Cue Reasoning

- 构建树状线索定位机制,实现结构化视觉线索感知。
- 动态奖励机制按推理需求调整,提升策略有效性。
- 自动生成数据集,适配监督微调与强化学习训练。
现有视频大语言模型在复杂视频理解上表现不足,推理能力有限且易产生幻觉,通常仅依赖预训练的内在推理逻辑,缺乏对输入视频内容的感知适应性。为此,我们提出视频树状线索推理(Video-ToC)框架,通过三项创新提升视频理解:(1) 树状引导的视觉线索定位机制,借助结构化推理模式增强模型细粒度感知能力;(2) 推理需求奖励机制,基于推理需求估计动态调整强化学习奖励值,实现按需激励更有效的推理策略;(3) 自动化标注流水线,构建 Video-ToC-SFT-1k 和 Video-ToC-RL-2k 数据集,分别用于监督微调和强化学习训练。在六个视频理解基准和一个视频幻觉基准上的大量评估表明,Video-ToC 显著优于基线及近期方法。代码已公开于 https://github.com/qizhongtan/Video-ToC。
原文摘要 · Abstract (English)
Existing Video Large Language Models (Video LLMs) struggle with complex video understanding, exhibiting limited reasoning capabilities and potential hallucinations. In particular, these methods tend to perform reasoning solely relying on the pretrained inherent reasoning rationales whilst lacking perception-aware adaptation to the input video content. To address this, we propose \textbf{Video-ToC}, a novel video reasoning framework that enhances video understanding through tree-of-cue reasoning. Specifically, our approach introduces three key innovations: (1) A tree-guided visual cue localization mechanism, which endows the model with enhanced fine-grained perceptual capabilities through structured reasoning patterns; (2) A reasoning-demand reward mechanism, which dynamically adjusts the reward value for reinforcement learning (RL) based on the estimation of reasoning demands, enabling on-demand incentives for more effective reasoning strategies; and (3) An automated annotation pipeline that constructs the Video-ToC-SFT-1k and Video-ToC-RL-2k datasets for supervised fine-tuning (SFT) and RL training, respectively. Extensive evaluations on six video understanding benchmarks and a video hallucination benchmark demonstrate the superiority of Video-ToC over baselines and recent methods. Code is available at https://github.com/qizhongtan/Video-ToC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。