解决超长视频推理中事件记忆断裂问题，通过因果事件结构增强模型理解能力。
Event-Causal RAG: A Retrieval-Augmented Generation Framework for Long Video Reasoning in Complex Scenarios
- 用视觉音频双哨兵机制分割视频为完整事件，构建状态-事件-状态因果结构。
- 在1小时以上视频上提升准确率4.96%~11.67%，支持持续流式处理。
- 适合需要长期上下文推理的视频分析任务，如监控、纪录片理解。
大型视觉语言模型在短中长视频理解上表现良好，但在超长视频中仍难以维持连贯的事件记忆并恢复长程关联。端到端方法受限于视觉令牌增长和上下文长度，而固定分段检索常导致事件碎片化，削弱状态转移建模。我们提出事件因果增强生成框架（EC-RAG），一种轻量级检索增强框架，用于超长及流式视频推理。双视觉-音频哨兵机制将视频流划分为语义完整的事件，以状态-事件-状态（SES）结构组织可观测前状态、核心事件与后状态，形成事件内因果转换。这些转换存储于双向量图内存中，并通过实体一致轨迹进行时间连接。问答时，双向图检索恢复相关前后事件，答案结合结构化记忆与对应视频证据生成。我们进一步引入ECV-1H，一个专为定向事件因果推理设计的一小时级长视频问答基准，所有源视频超过一小时，覆盖超过150小时未剪辑视频，包含1,251对全人工标注的问答对。EC-RAG在三个开源视频基础模型上整体准确率提升4.96%–11.67%，并在多个公开数据集上实现稳定增益。在单张32GB显存的RTX 5090 GPU上，可连续处理视频并控制流式内存使用。
原文摘要 · Abstract (English)
Large vision-language models perform well on short- and medium-length video understanding but still struggle to maintain coherent event memory and recover long-range relationships in ultra-long videos. End-to-end methods are limited by visual-token growth and context length, while fixed-segment retrieval often fragments complete events and weakens state-transition modeling.We propose Event-Causal RAG (EC-RAG), a lightweight retrieval-augmented framework for ultra-long and streaming video reasoning. A dual visual-audio sentinel mechanism segments video streams into semantically complete events, represented as State-Event-State (SES) structures that organize observable pre-event states, central events, and post-event states as event-local causal transitions. These transitions are stored in dual vector-graph memory and temporally connected through entity-consistent trajectories. During question answering, bidirectional graph retrieval recovers relevant predecessor and successor events, and answers are generated using both structured memory and the corresponding video evidence.We further introduce ECV-1H, an hour-scale long-video QA benchmark dedicated to directed event-causal reasoning, with all source videos exceeding one hour. It covers over 150 hours of untrimmed video and contains 1,251 fully human-annotated QA pairs. EC-RAG improves overall accuracy by 4.96\%--11.67\% across three open-source video foundation models and achieves consistent gains across public datasets. On a single RTX 5090 GPU with 32 GB of memory, EC-RAG can continuously process videos while maintaining controlled streaming memory usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。