根据问题复杂度自适应选择检索策略,提升长视频理解效率与准确率。
AdaVideoRAG: Omni-Contextual Adaptive Retrieval-Augmented Efficient Long Video Understanding
- 通过轻量级意图分类器动态选择检索方案,从简单到复杂逐级匹配
- 在长视频问答任务中显著提升准确率,同时降低冗余计算和延迟
- 适用于需要深度推理的长视频分析场景,可无缝接入现有多模态大模型
多模态大语言模型在视频理解中表现良好,但在长视频上因固定上下文长度和弱长期依赖建模而性能下降。检索增强生成(RAG)可动态扩展知识,但现有视频RAG方案采用固定检索模式,忽略查询难度。这种统一设计导致简单查询产生冗余计算和延迟,复杂多跳推理则因粗粒度检索遗漏关键信息。单步检索严重限制了效率与认知深度之间的平衡。我们提出AdaVideoRAG,一种面向长视频理解的自适应RAG框架。轻量级意图分类器根据查询复杂度动态选择从最简单到最复杂的检索方案。设计了全知识索引模块,将多模态信息组织为三个数据库:(1) 基于片段字幕、ASR和OCR构建的文本库;(2) 视觉库;(3) 支持深层语义理解的知识图谱。支持从基础检索到图谱检索的分层知识访问,平衡资源开销与推理能力。为评估深度理解能力,进一步构建了HiVU基准测试。实验表明,AdaVideoRAG在长视频问答任务中显著提升效率与准确率,可通过轻量级API无缝集成至现有MLLMs,确立了自适应检索增强视频分析的新范式。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) perform well in video understanding but degrade on long videos due to fixed-length context and weak long-term dependency modeling. Retrieval-Augmented Generation (RAG) can expand knowledge dynamically, yet existing video RAG schemes adopt fixed retrieval paradigms that ignore query difficulty. This uniform design causes redundant computation and latency for simple queries, while coarse retrieval for complex, multi-hop reasoning can miss key information. Such single-step retrieval severely limits the trade-off between efficiency and cognitive depth. We propose AdaVideoRAG, an adaptive RAG framework for long-video understanding. A lightweight intent classifier dynamically selects suitable retrieval schemes according to query complexity from the simplest to the most sophisticated. We design an Omni-Knowledge Indexing module that extracts and organizes multi-modal information into three databases: (1) a text base built from clip captions, ASR, and OCR; (2) a visual base; and (3) a knowledge graph for deep semantic understanding. This supports hierarchical knowledge access, from naive retrieval to graph-based retrieval, balancing resource cost and reasoning ability. To evaluate deep understanding, we further construct the HiVU benchmark. Experiments show that AdaVideoRAG significantly improves both efficiency and accuracy on long-video QA tasks and can be seamlessly plugged into existing MLLMs through lightweight APIs, establishing a new paradigm for adaptive retrieval-augmented video analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。