通过持续提取与自适应查询,提升长视频问答的语义理解与细节检索能力。
CAM: Question Answering on Entity-Centric Videos with Continuous Extraction and Adaptive Querying

- 基于知识图谱持续聚合实体关系,捕捉长期语义特征。
- 支持多模态搜索方法,精准定位问答所需细粒度信息。
- 在三个基准上提升准确率最高达23个百分点,适合长视频理解任务。
记忆机制通过提取并检索事实,帮助多模态大模型(MLLMs)在长视频上进行问答,克服其有限上下文窗口的限制。现有方法通常从固定时长视频片段中提取独立的记忆条目,难以捕捉需跨长时间总结的高层语义(如角色特质和关系)。此外,仅依赖相似性检索可能遗漏问答所需的细粒度信息。为此,本文提出CAM,具备连续提取以捕获高层语义、自适应查询以获取细粒度细节的能力。CAM将视频片段中提取的实体与关系存入知识图谱,当目标实体或关系的局部子图达到预设规模时,对其进行汇总以捕捉高层语义。为检索问答所需细粒度信息,CAM支持知识图谱遍历、视频重看和音频听觉等多种搜索方式,并通过规划-执行-验证流水线,根据问题意图动态组合这些方法。在三个基准上的评估表明,CAM优于当前最优基线,准确率最高提升23个百分点。代码已开源:https://github.com/Jake-Tian/CAM。
原文摘要 · Abstract (English)
Memory facilitates question answering over long videos by extracting and retrieving facts to fit within the limited context windows of multimodal LLMs (MLLMs). Existing solutions typically extract independent memory entries from fixed-length video clips and thus cannot capture high-level semantics that need to be summarized over extended time periods, such as character traits and relations. Moreover, they rely solely on similarity-based retrieval and may fail to retrieve the fine-grained details required for question answering. To tackle these problems, we propose CAM, featuring continuous extraction for high-level semantics and adaptive querying for fine-grained details. In particular, CAM stores the entities and relations extracted from video clips in a knowledge graph. To capture the high-level semantics of each entity or relation, CAM summarizes the local subgraph of the target entity or relation once the subgraph reaches a predefined size. To retrieve the fine-grained details required for question answering, CAM supports multiple search methods, including knowledge graph traversal, video re-watching, and audio listening. It utilizes a planner-executor-verifier pipeline to adaptively compose these search methods according to question intent. Evaluations on three benchmarks show that CAM outperforms SOTA baselines and improves their accuracy by up to 23 percentage points. Code is available at https://github.com/Jake-Tian/CAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。