arXiv:2603.15167cs.CV2026-03中稿 · CVPR被引 1

让视频理解模型通过记忆反馈,更准确地回答长期事件问题。

Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding

  • 用问题引导注意力,融合当前帧与记忆中的历史帧信息
  • 在多个长视频任务上提升显著,最高达18.3%的性能增益
  • 适合需要长期上下文理解的视频分析场景

在大型多模态模型进行长期视频理解的背景下,尽管基于Transformer的视觉压缩器和带记忆的方法已被广泛使用,但它们通常独立压缩每一帧,难以有效处理需理解完整事件的任务,如MLVU和VNBench中的时序排序任务。为此,我们重新思考从感知到记忆的传统单向流程,提出一种由问题引导、具备记忆反馈的视觉压缩框架QViC-MF。其核心是问题引导的多模态选择性注意力(QMSA),可从当前片段及记忆中过去的相关帧中保留与问题相关的视觉信息。压缩器与记忆反馈对全视频每个片段迭代运行。该简单而有效的设计在长期视频理解任务中带来显著性能提升:在MLVU测试集上比当前最佳方法提升6.1%,在LVBench上提升8.3%,在VNBench Long上提升18.3%,在VideoMME Long上提升3.7%。代码将公开发布。

原文摘要 · Abstract (English)

In the context of long-term video understanding with large multimodal models, many frameworks have been proposed. Although transformer-based visual compressors and memory-augmented approaches are often used to process long videos, they usually compress each frame independently and therefore fail to achieve strong performance on tasks that require understanding complete events, such as temporal ordering tasks in MLVU and VNBench. This motivates us to rethink the conventional one-way scheme from perception to memory, and instead establish a feedbackdriven process in which past visual contexts stored in the context memory can benefit ongoing perception. To this end, we propose Question-guided Visual Compression with Memory Feedback (QViC-MF), a framework for long-term video understanding. At its core is a Question-guided Multimodal Selective Attention (QMSA), which learns to preserve visual information related to the given question from both the current clip and the past related frames from the memory. The compressor and memory feedback work iteratively for each clip of the entire video. This simple yet effective design yields large performance gains on longterm video understanding tasks. Extensive experiments show that our method achieves significant improvement over current state-of-the-art methods by 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long. The code will be released publicly.

视频理解记忆机制注意力机制长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。