arXiv:2508.01546cs.CV2025-08被引 12

E-VRAG通过轻量级检索提升长视频理解效率,计算成本降70%。

E-VRAG: Enhancing Long Video Understanding with Resource-Efficient Retrieval Augmented Generation

  • 分层查询分解预过滤帧,从数据端降低冗余计算。
  • 轻量VLM评分+全局分数分布策略,兼顾效率与准确率。
  • 多视角问答增强信息提取,适合长视频分析场景。

视觉语言模型(VLMs)通过跨模态推理推动了视频理解的发展,但受限于有限的上下文窗口和处理千帧以上长视频的高计算成本。检索增强生成(RAG)通过仅选择最相关帧输入,减轻计算负担。然而现有视频RAG方法难以平衡检索效率与准确性,尤其在复杂多样的视频内容上表现不佳。为此,我们提出E-VRAG,一种高效视频RAG框架。首先基于分层查询分解的帧预过滤方法,在数据层面剔除无关帧,降低计算开销;其次采用轻量VLM进行帧评分,在模型层面进一步减少资源消耗;同时提出利用帧间评分全局统计分布的检索策略,缓解轻量VLM带来的性能下降问题;最后引入多视角问题回答机制,增强VLM对长视频上下文的信息提取与理解能力。在四个公开基准上的实验表明,E-VRAG相比基线方法实现约70%的计算成本降低,且精度更高,且无需额外训练。结果验证了其在提升视频RAG任务效率与准确性的有效性。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) have enabled substantial progress in video understanding by leveraging cross-modal reasoning capabilities. However, their effectiveness is limited by the restricted context window and the high computational cost required to process long videos with thousands of frames. Retrieval-augmented generation (RAG) addresses this challenge by selecting only the most relevant frames as input, thereby reducing the computational burden. Nevertheless, existing video RAG methods struggle to balance retrieval efficiency and accuracy, particularly when handling diverse and complex video content. To address these limitations, we propose E-VRAG, a novel and efficient video RAG framework for video understanding. We first apply a frame pre-filtering method based on hierarchical query decomposition to eliminate irrelevant frames, reducing computational costs at the data level. We then employ a lightweight VLM for frame scoring, further reducing computational costs at the model level. Additionally, we propose a frame retrieval strategy that leverages the global statistical distribution of inter-frame scores to mitigate the potential performance degradation from using a lightweight VLM. Finally, we introduce a multi-view question answering scheme for the retrieved frames, enhancing the VLM's capability to extract and comprehend information from long video contexts. Experiments on four public benchmarks show that E-VRAG achieves about 70% reduction in computational cost and higher accuracy compared to baseline methods, all without additional training. These results demonstrate the effectiveness of E-VRAG in improving both efficiency and accuracy for video RAG tasks.

视频理解RAG轻量化长视频

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。