用图文混合查询实现监控视频精准定位事件,提升刑侦搜索效率。
ForeSea: AI Forensic Search with Multi-modal Queries for Video Surveillance
- 分三阶段构建可插拔的视频检索系统,先过滤后索引再推理。
- 在复杂图文查询下,准确率提升3.1点,时间定位精度提高10.1点。
- 首个支持精确时间锚定的多模态视频问答基准,适合刑侦与安防场景。
尽管多年研究,监控视频搜索与推理仍面临挑战:长时序、多摄像头环境下对特定目标的追踪困难。现有方法如跟踪、检索和视频大模型依赖大量人工筛选,仅能捕捉浅层特征,缺乏时间理解能力。已有基准也仅限于基础检索与问答,未覆盖真实场景中常见的多模态查询与时间定位需求(如“这个人何时加入打斗?”并附其图像)。为此,我们提出ForeSeaQA,一个专为带图像与文本查询的视频问答设计的新基准,包含长时间监控视频与多样化的多模态问题,并标注关键事件的时间戳。该数据集支持在真实刑侦条件下系统评估检索、时间定位与多模态推理能力。基于此,我们构建ForeSea——一个三阶段可插拔的AI刑侦搜索系统:(1) 跟踪模块过滤无关视频;(2) 多模态嵌入模块索引剩余片段;(3) 推理时检索前K个候选片段供视频大模型回答问题并定位事件。在ForeSeaQA基准上,ForeSea相比先前检索增强基线,准确率提升3.1点,时间交并比(IoU)提升10.1点。据我们所知,ForeSeaQA是首个支持复杂多模态查询与精确时间定位的基准,ForeSea则是首个为此类任务量身打造的VideoRAG系统。
原文摘要 · Abstract (English)
Despite decades of work, surveillance still struggles in searching and reasoning about specific targets across long, multi-camera videos. Existing methods - tracking, retrieval, and video LLMs require heavy manual filtering, capture only shallow attributes, and fail at temporal understanding. Prior benchmarks are also limited to basic retrieval and question answering, without addressing real world challenges that often involve multimodal queries and temporal grounding (e.g., "When did this person join the fight?" with the person's image). To address this gap, we introduce ForeSeaQA, a new benchmark specifically designed for video QA with image-and-text queries and timestamped annotations of key events. The dataset consists of long-horizon surveillance footage paired with diverse multimodal questions, enabling systematic evaluation of retrieval, temporal grounding, and multimodal reasoning in realistic forensic conditions. Not limited to this benchmark, we propose ForeSea, an AI forensic search system with a 3-stage, plug-and-play pipeline. (1) A tracking module filters irrelevant footage; (2) a multimodal embedding module indexes the remaining clips; and (3) during inference, the system retrieves top-K candidate clips for a video LLM to answer queries and localize events. On ForeSeaQA benchmark, ForeSea improves accuracy by 3.1 points and temporal IoU by 10.1 points over prior retrieval-augmented baselines. To our knowledge, ForeSeaQA is the first benchmark to support complex multimodal queries with precise temporal grounding, and ForeSea is the first VideoRAG system built to excel in this setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。