arXiv:2509.24304cs.CV2025-09被引 29

让大模型像人一样思考长视频,只看关键帧就能答对难题。

FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting

  • 通过多轮选帧提问,让模型主动聚焦视频重点
  • 7B模型仅用20.6帧达76.1%准确率,比基线快20倍
  • 适合需要高效长视频理解的AI系统研发者

尽管大型视觉语言模型(LVLMs)在视频理解方面取得显著进展,但其在长视频推理中的应用受限于均匀采样和静态文本推理,效率低且难以应对视觉密集型任务。为此,本文提出框架FrameThinker,使LVLM能够迭代式地追问视频内容。该框架面临两大挑战:适应新动作(如选帧)及设计引导策略的奖励函数。为此,我们采用两阶段训练:先通过监督微调(SFT)赋予基础动作能力,再通过强化学习(RL)优化决策策略。在视频推理基准(Video-Holmes、LongVideo-Reason)和长视频理解数据集(LongVideoBench、MLVU、VideoMME、LVBench)上的实验表明,FrameThinker平均性能提升10.4%,同时大幅减少处理帧数。特别地,7B模型在LongVideo-Reason上达到76.1%准确率,平均仅使用20.6帧,优于竞争模型LongVILA-R1(72.0%),且帧数仅为512的1/20,展现出卓越的效率与效果。

原文摘要 · Abstract (English)

While Large Vision-Language Models (LVLMs) have achieved substantial progress in video understanding, their application to long video reasoning is hindered by uniform frame sampling and static textual reasoning, which are inefficient and struggle to handle visually intensive video tasks. To overcome these challenges, in this paper, we introduce the concept of thinking with long videos and propose a novel framework FrameThinker. Within this framework, LVLMs are able to iteratively interrogate video content. Developing such video reasoning capabilities in LVLMs presents notable challenges, particularly in adapting the model to new video actions (e.g. select frame), and designing reward functions to guide LVLMs to adopt the newly introduced action. To solve these challenges, we propose a two-phase training strategy, first employing Supervised Fine-Tuning (SFT) to instill fundamental action capabilities, followed by Reinforcement Learning (RL) to optimize a strategic decision-making policy. Notably, in this RL phase, we conduct an in-depth and comprehensive exploration of the reward design for each action and format reward. Extensive experiments on reasoning benchmarks like Video-Holmes, LongVideo-Reason, and long-video understanding benchmarks such as LongVideoBench, MLVU, VideoMME, and LVBench, demonstrate that FrameThinker achieves a significant average improvement of +10.4% over baselines while drastically reducing the number of processed frames. Most notably, our 7B model, FrameThinker establishes a new state-of-the-art on LongVideo-Reason, achieving 76.1% accuracy using an average of only 20.6 frames. This not only outperforms the competitive LongVILA-R1 (72.0%) but does so with over 20x fewer frames (vs. 512), demonstrating unparalleled efficiency and effectiveness.

长视频理解视觉推理强化学习高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。