通过注意力筛选视频帧，实现高效流式视频理解。
Recurrent Attention-based Token Selection for Efficient Streaming Video-LLMs
- 基于大模型注意力选择关键视觉帧，丢弃近95%无关帧。
- 循环处理已选帧，保持时序连贯性，性能损失小。
- 用自动生成字幕回答问题，轻量且准确，适合实时场景。
视频大语言模型（Video-LLMs）在具备完整视频访问权限时能出色理解视频内容。但在流式场景中，需对长达一小时的视频进行在线处理，并及时响应问题，现有方法面临挑战。本文提出一种无需训练的兼容方案，核心包含三方面：1）利用大模型自身注意力机制识别对每个短片段理解有贡献的关键视觉帧，实现注意力驱动的帧选择，可丢弃约95%的不重要帧而性能损失极小；2）对历史选出的帧进行递归处理，构建每个片段的时序连贯理解；3）基于生成字幕进行问答，实现轻量化且高精度响应。该方法在流式视频基准测试中达到当前最优表现，在效率与效果间取得良好平衡。
原文摘要 · Abstract (English)
Video Large Language Models (Video-LLMs) excel at understanding videos in-context, provided they have full access to the video when answering queries. However, these models face challenges in streaming scenarios where hour-long videos must be processed online, and questions need timely responses. In this work, we propose a training-free approach compatible with standard Video-LLMs, leveraging three key concepts: 1) LLM-informed selection of visual tokens to identify those that the LLM has attended to and contributed to its understanding of each short clip. Our attention-based selection allows us to discard up to ~95% of unimportant visual tokens with minimal performance loss; 2) Recurrent processing of past selected tokens to generate temporally coherent understanding of each processed clip; 3) Caption-based question answering for lightweight and accurate responses. Our method achieves state-of-the-art performance on streaming video benchmarks, striking a balance between efficiency and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。