arXiv:2503.09590cs.CV2025-03CVPR被引 28

提出BIMBA模型,高效压缩长视频冗余信息以提升问答准确率。

BIMBA: Selective-Scan Compression for Long-Range Video Question Answering

  • 用选择性扫描算法筛选关键视频帧,减少冗余信息。
  • 在6个基准测试上达到当前最佳性能,最高提升5.2%准确率。
  • 适合需要处理长视频的多模态大模型研究者使用。

长视频问答(VQA)的核心挑战在于从大量冗余帧中提取相关信息并建模长距离依赖。自注意力机制虽具通用性,但处理长视频中的海量时空标记时计算成本过高。现有方法多通过稀疏采样或时空池化压缩输入/输出序列,但常过度保留冗余内容,遗漏关键事件或快速变化的时空模式。本文提出BIMBA,一种高效的状态空间模型,利用选择性扫描算法从高维视频中学习选择关键信息,并转换为精简的标记序列以供大语言模型高效处理。大量实验证明,BIMBA在多个长视频VQA基准上取得最优表现,涵盖PerceptionTest、NExT-QA、EgoSchema、VNBench、LongVideoBench和Video-MME。代码与模型已公开于https://sites.google.com/view/bimba-mllm。

原文摘要 · Abstract (English)

Video Question Answering (VQA) in long videos poses the key challenge of extracting relevant information and modeling long-range dependencies from many redundant frames. The self-attention mechanism provides a general solution for sequence modeling, but it has a prohibitive cost when applied to a massive number of spatiotemporal tokens in long videos. Most prior methods rely on compression strategies to lower the computational cost, such as reducing the input length via sparse frame sampling or compressing the output sequence passed to the large language model (LLM) via space-time pooling. However, these naive approaches over-represent redundant information and often miss salient events or fast-occurring space-time patterns. In this work, we introduce BIMBA, an efficient state-space model to handle long-form videos. Our model leverages the selective scan algorithm to learn to effectively select critical information from high-dimensional video and transform it into a reduced token sequence for efficient LLM processing. Extensive experiments demonstrate that BIMBA achieves state-of-the-art accuracy on multiple long-form VQA benchmarks, including PerceptionTest, NExT-QA, EgoSchema, VNBench, LongVideoBench, and Video-MME. Code, and models are publicly available at https://sites.google.com/view/bimba-mllm.

视频问答长视频状态空间模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。