arXiv:2607.16189cs.CV2026-07

用可自修正的树状搜索提升长视频问答精准度。

Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA

论文配图:Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA
图 1 · 摘自论文原文
  • 构建动态时间树,通过缩放、移动等操作实现精细定位与回溯。
  • 在三个基准上显著超越现有方法,最高提升7.4分准确率。
  • 适合需要精准定位与纠错能力的长视频问答任务。

基于长视频的问题回答(Grounded LVQA)需在长视频中定位支持答案的短片段。现有代理方法虽能粗略到精细地探索,但缺乏从细到粗回溯的能力,常过早收敛且无法纠正错误。本文提出VideoTreeSearch(VTS),将该任务建模为在自适应时间树上的迭代自修正搜索。该树基于视觉场景边界生成,每个节点对应一个语义连贯片段,代理通过zoom_in、zoom_out、shift和answer四种离散操作导航。我们设计轨迹合成管道生成多步路径,包含故意进入错误分支后恢复的过程,用于监督微调与强化学习,奖励包括定位精度和答案正确性。在CG-Bench、Haystack-LVBench和Haystack-Ego4D三个基准上,VTS分别取得+12.5 mIoU和+7.4 T-F1的提升。迁移至通用长视频问答任务时,在Video-MME、MLVU和LVBench上最高提升7.1分。消融实验证明,自修正的层级搜索是性能关键:移除自适应下探或显式回溯均导致性能大幅下降。代码已开源。

原文摘要 · Abstract (English)

Grounded long-video question answering (Grounded LVQA) requires answering a question about a long video while localizing the short evidence interval that supports the answer. Recent agentic methods frame this task as multi-turn exploration with a single crop_video(start, end) action, which supports coarse-to-fine narrowing but provides no primitive for fine-to-coarse backtracking. As a result, these agents typically converge prematurely and cannot recover from an early mistake. We propose VideoTreeSearch (VTS), a framework that casts grounded LVQA as iterative self-correcting search over an adaptive temporal tree. VTS constructs a non-uniform tree from visual scene boundaries so that each node corresponds to a semantically coherent segment, and trains an agent to navigate the tree through four discrete operations: zoom_in, zoom_out, shift, and answer. These operations expose backtracking and recovery as explicit, learnable primitives rather than implicit behaviors. To train this navigation, we introduce a trajectory synthesis pipeline that produces multi-step paths through the tree, including deliberate detours into incorrect branches followed by recovery. We use these trajectories for supervised fine-tuning, followed by reinforcement learning with grounding and answer-accuracy rewards. On three Grounded LVQA benchmarks (CG-Bench, Haystack-LVBench, Haystack-Ego4D), VTS outperforms the strongest prior agentic methods by +12.5 mIoU on CG-Bench and +7.4 T-F1 on Haystack-Ego4D. The learned policy also transfers to general long-video QA, surpassing all prior agentic baselines on Video-MME, MLVU, and LVBench by up to +7.1 accuracy points. Ablations confirm that self-correcting hierarchical search is the central mechanism behind these gains: removing either adaptive descent or explicit backtracking substantially degrades performance. Code is available at https://github.com/CeeZh/VTS.

视频问答自修正树状搜索长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。