arXiv:2605.08735cs.CV2026-05被引 3

用视觉语言模型与视频生成模型协同推理,提升长时任务准确性

CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

论文配图:CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models
图 1 · 摘自论文原文
  • 在每一步中让视觉语言模型规划动作并检查生成视频片段
  • 在相同计算量下比单次推理和现有方法在难任务上提升显著
  • 适合需要多步推理的视频理解任务,尤其对复杂长时任务有效

近期的“以视频思考”方法利用视频生成模型(VGMs)通过生成时间连贯的帧序列作为推理产物。然而,即使强大的VGMs在目标导向任务中仍存在两种常见失败模式:多步骤任务中的长时程漂移,以及中段片段模拟误差累积。这两种问题均源于VGM短时视觉先验缺乏显式推理支持,而这一角色本应由视觉语言模型(VLMs)承担。但如何放置VLM具有挑战性:前期规划在生成任何帧前就锁定策略,后期批判则对完整视频进行干预过晚。为此,我们提出VLM-VGM协同视频推理(CollabVR),一种闭环框架,将VLM与VGM以步骤级粒度耦合:VLM规划下一步动作,检查VGM生成的片段,并将验证结果直接反馈至下一动作提示中以修复错误。在Gen-ViRe和VBVR-Bench数据集上,CollabVR在匹配计算成本下优于开源与闭源VGM,超越单次推理、Pass@$k$及先前测试时缩放基线,尤其在最难任务上提升最大。此外,其在经过推理微调的VGM基础上仍有进一步增益,表明步骤级VLM监督与推理微调具有正交性且可叠加。视频样例与更多定性结果见项目页面:https://joow0n-kim.github.io/collabvr-project-page。

原文摘要 · Abstract (English)

Recent "Thinking with Video" approaches use Video Generation Models (VGMs) for visual reasoning by producing temporally coherent Chain-of-Frames as reasoning artifacts. Even strong VGMs, however, exhibit two recurring failure modes on goal-directed tasks: long-horizon drift on multi-step tasks and mid-clip simulation errors that compound. Both stem from the absence of explicit reasoning built upon the VGM's short-horizon visual prior, a role naturally filled by Vision-Language Models (VLMs), but where to place the VLM is non-trivial: upfront plans commit before any frame is generated and post-hoc critiques over whole videos intervene too late. We propose VLM-VGM Collaborative Video Reasoning (CollabVR), a closed-loop framework that couples the VLM with the VGM at step-level granularity: the VLM plans the immediate next action, inspects the clip the VGM generates, and folds the verifier's diagnosis directly into the next action prompt to repair detected failures. On Gen-ViRe and VBVR-Bench, CollabVR improves both open-source and closed-source VGMs over single-inference, Pass@$k$, and prior test-time scaling baselines at matched compute, with the largest gains on the hardest tasks. It also yields further improvements on top of a reasoning-fine-tuned VGM, indicating that step-level VLM supervision is orthogonal to and stackable with reasoning-oriented fine-tuning. We provide video samples and additional qualitative results at our project page: https://joow0n-kim.github.io/collabvr-project-page.

视频推理多模态协同建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。