arXiv:2606.24477cs.CVcs.AI2026-06

让视频大模型学会反复观看、提问和修正答案,更高效准确地理解视频。

video-SALMONN-R$^3$: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding

论文配图:video-SALMONN-R$^3$: Learning to ReWatch, ReAsk, and ReAnswer for Efficient Video Understanding
图 1 · 摘自论文原文
  • 通过强化学习实现端到端的反复观看,无需昂贵的思维链标注。
  • 在多个数据集上超越基线模型,且计算成本更低。
  • 适合追求高精度视频理解且资源受限的应用场景。

视频大语言模型常受算力与内存限制,导致采样帧率和空间分辨率降低,可能遗漏问答关键信息。本文提出 video-SALMONN-R$^3$,首个无需依赖思维链(CoT)冷启动的端到端视频-大模型,通过强化学习实现重看机制。为解决重看引发的推理优先行为与预训练模型的答案优先倾向之间的不匹配,提出重答策略:首次观看直接生成答案,重看后进行修正。为提升重看时的问题遵循度,引入重问机制,在回看定位片段时重新注入原始问题。实验表明,video-SALMONN-R$^3$ 在多个视频问答数据集上均优于基线模型及现有重看方法,且计算开销显著更低。代码、模型与数据将在录用后公开。

原文摘要 · Abstract (English)

Video large language models (LLMs) are often constrained by computation and memory budgets, leading them to use reduced frame rates and spatial resolutions, which may cause them to miss critical information for question answering (QA). A practical and efficient solution is a two-stage paradigm: first perform coarse video understanding to localize relevant segments, and then re-watch these segments at higher temporal or spatial fidelity. In this paper, we present video-SALMONN-R$^3$, the first end-to-end video-LLM that enables re-watch through reinforcement learning without relying on chain-of-thought (CoT) cold-start. This design removes the need for costly CoT data annotations and avoids CoT-based supervised fine-tuning (SFT), which can otherwise degrade the pretrained video understanding abilities. To address the mismatch between the reasoning-first behavior induced by re-watch and the answer-first tendency of pretrained video-LLMs, we propose a re-answer strategy, in which the model first produces a direct answer in the first watch and then refines it after re-watching. Finally, to improve question adherence during re-watching, we propose a re-ask mechanism that re-injects the query when revisiting localized segments. Experimental results show that video-SALMONN-R$^3$ consistently outperforms both the base model and the QA-SFT baseline, while surpassing prior re-watch-based approaches with significantly lower computational cost. Code, models, and data will be publicly released upon acceptance.

视频理解大模型重看机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。