首个可处理超长视频的检索增强生成框架,让大模型读懂小时级视频内容。
VideoRAG: Retrieval-Augmented Generation with Extreme Long-Context Videos
- 双通道设计:图文知识图谱+多模态编码,跨视频建模语义关系。
- 在134小时视频上表现超越现有方法,支持无限长度视频理解。
- 适合需要长视频分析的研究者与开发者,开源代码已发布。
检索增强生成(RAG)通过引入外部知识显著提升了大语言模型的能力,但其应用主要局限于文本,对多模态视频知识的利用仍处于空白。本文提出VideoRAG,首个专为处理极长视频设计的检索增强生成框架。核心创新在于双通道架构:(i)基于图的文本知识定位,捕捉跨视频语义关系;(ii)多模态上下文编码,高效保留视觉特征。该设计使VideoRAG能通过构建跨视频的精准知识图谱,结合专用多模态检索机制,实现无限制长度视频的理解。在自研LongerVideos基准上进行综合评估,该数据集包含超过160个视频,总时长134+小时,涵盖讲座、纪录片和娱乐类内容。实验表明,VideoRAG在性能上显著优于现有RAG方案及长视频理解方法。VideoRAG的源码与基准数据集已公开:https://github.com/HKUDS/VideoRAG。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has demonstrated remarkable success in enhancing Large Language Models (LLMs) through external knowledge integration, yet its application has primarily focused on textual content, leaving the rich domain of multi-modal video knowledge predominantly unexplored. This paper introduces VideoRAG, the first retrieval-augmented generation framework specifically designed for processing and understanding extremely long-context videos. Our core innovation lies in its dual-channel architecture that seamlessly integrates (i) graph-based textual knowledge grounding for capturing cross-video semantic relationships, and (ii) multi-modal context encoding for efficiently preserving visual features. This novel design empowers VideoRAG to process unlimited-length videos by constructing precise knowledge graphs that span multiple videos while maintaining semantic dependencies through specialized multi-modal retrieval paradigms. Through comprehensive empirical evaluation on our proposed LongerVideos benchmark-comprising over 160 videos totaling 134+ hours across lecture, documentary, and entertainment categories-VideoRAG demonstrates substantial performance compared to existing RAG alternatives and long video understanding methods. The source code of VideoRAG implementation and the benchmark dataset are openly available at: https://github.com/HKUDS/VideoRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。