提出自适应推理框架,让模型快慢结合处理长视频问答。
VideoPro: Adaptive Program Reasoning for Long Video Understanding
- 设计快慢双轨推理机制,简单问题快速回答,难题触发深度分析。
- 在LVBench上达50.4%准确率,超越GPT-4o,接近顶尖模型性能。
- 适合需要高效可靠长视频理解的场景,如智能客服、内容审核。
大语言模型在生成视觉任务程序流程方面展现出潜力。然而,以往方法多依赖闭源模型,缺乏系统性推理能力,且难以应对长视频问答任务。为此,我们提出FS-VisPR框架,一种自适应视觉程序推理方法,可在简单查询时快速推理,在复杂问题时切换至慢速推理。首先,设计高效视觉模块(如关键片段检索与字幕检索)以支持长视频任务;其次,利用强语言模型构建多样且高质量的快慢推理数据集,对齐开源语言模型生成视觉程序流程的能力,形成FS-LLM。随后,设计快慢推理框架:简单问题由VideoLLM直接解答,复杂问题则启动视觉程序推理,模拟人类思维过程。低置信度的快速答案将触发第二阶段慢速推理,若程序执行失败,则回退至快速推理。此外,通过训练和推理阶段的参数搜索优化视觉程序:调整程序内视觉模块参数生成多个变体,训练时选择正确答案的程序,推理时采用最高置信度结果。实验表明,该框架在视觉程序流程中显著提升效率与可靠性,在LVBench上达到50.4%准确率,超越GPT-4o,与Qwen2.5VL-72B在VideoMME上的表现相当。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown promise in generating program workflows for visual tasks. However, previous approaches often rely on closed-source models, lack systematic reasoning, and struggle with long-form video question answering (videoQA). To address these challenges, we introduce the FS-VisPR framework, an adaptive visual program reasoning approach that balances fast reasoning for simple queries with slow reasoning for difficult ones. First, we design efficient visual modules (e.g., key clip retrieval and subtitle retrieval) to support long-form video tasks. Then, we construct a diverse and high-quality fast-slow reasoning dataset with a strong LLM to align open-source language models' ability to generate visual program workflows as FS-LLM. Next, we design a fast-slow reasoning framework with FS-LLM: Simple queries are directly solved by VideoLLMs, while difficult ones invoke visual program reasoning, motivated by human-like reasoning processes. During this process, low-confidence fast-thinking answers will trigger a second-stage slow-reasoning process, and a fallback mechanism to fast reasoning is activated if the program execution fails. Moreover, we improve visual programs through parameter search during both training and inference. By adjusting the parameters of the visual modules within the program, multiple variants are generated: during training, programs that yield correct answers are selected, while during inference, the program with the highest confidence result is applied. Experiments show that FS-VisPR improves both efficiency and reliability in visual program workflows. It achieves 50.4% accuracy on LVBench, surpassing GPT-4o, matching the performance of Qwen2.5VL-72B on VideoMME.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。