用场景分割提升长视频理解，让AI更像人一样看懂连续剧情。
SceneRAG: Scene-level Retrieval-Augmented Generation for Video Understanding
- 基于语音转录与时间元数据，自动划分视频为连贯叙事场景。
- 在134小时视频上生成任务胜率达72.5%，显著优于已有方法。
- 适合需要理解长视频上下文的场景，如纪录片、会议记录分析。
尽管检索增强生成（RAG）在视频理解中取得进展，但长视频内容的高效理解仍因数据规模大、结构复杂而未被充分探索。现有RAG方法通常将视频切分为固定长度片段，常破坏上下文连续性，难以捕捉真实场景边界。受人类自然将连续体验划分为连贯场景的能力启发，我们提出SceneRAG，一种统一框架，利用大语言模型结合语音识别（ASR）转录文本与时间元数据，对视频进行叙事一致的场景分割。该框架通过轻量启发式规则和迭代修正进一步优化初始边界。针对每个场景，融合视觉与文本模态信息，提取实体关系并动态构建知识图谱，支持鲁棒的多跳检索与生成，以处理长程依赖。在包含超过134小时多样化内容的LongerVideos基准上实验表明，SceneRAG显著优于先前基线，在生成任务中最高实现72.5%的胜率。
原文摘要 · Abstract (English)
Despite recent advances in retrieval-augmented generation (RAG) for video understanding, effectively understanding long-form video content remains underexplored due to the vast scale and high complexity of video data. Current RAG approaches typically segment videos into fixed-length chunks, which often disrupts the continuity of contextual information and fails to capture authentic scene boundaries. Inspired by the human ability to naturally organize continuous experiences into coherent scenes, we present SceneRAG, a unified framework that leverages large language models to segment videos into narrative-consistent scenes by processing ASR transcripts alongside temporal metadata. SceneRAG further sharpens these initial boundaries through lightweight heuristics and iterative correction. For each scene, the framework fuses information from both visual and textual modalities to extract entity relations and dynamically builds a knowledge graph, enabling robust multi-hop retrieval and generation that account for long-range dependencies. Experiments on the LongerVideos benchmark, featuring over 134 hours of diverse content, confirm that SceneRAG substantially outperforms prior baselines, achieving a win rate of up to 72.5 percent on generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。