让大模型理解长视频,靠结构化检索和意图感知。
VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAG

- 构建视频时空图,多跳检索跨段落关联信息
- 引入意图评分器,精准找回与问题逻辑匹配的帧
- 自建60万样本数据集,支持意图对齐训练
将多模态大模型应用于长视频时,受限于有限的上下文窗口。虽然检索增强生成(RAG)可通过组织相关视觉证据压缩上下文来缓解此问题,但现有方法存在两大缺陷:(i) 将视频切分为独立片段,破坏其固有的时空结构;(ii) 依赖显式语义匹配,可能遗漏与查询意图隐含相关的线索。为此,我们提出VideoStir,一种结构化且意图感知的长视频RAG框架。该框架首先在片段层面将视频建模为时空图,然后执行多跳检索,聚合远距离但语境相关的事件证据。此外,引入基于多模态大模型的意图相关性评分器,依据帧与查询推理意图的一致性进行检索。为支持该能力,我们构建了IR-600K,一个大规模数据集,专用于学习帧与查询意图的对齐关系。实验表明,VideoStir在不依赖辅助信息的情况下,性能媲美当前最优基线,凸显了从扁平化语义匹配转向结构化、意图感知推理的潜力。代码与模型权重已公开于https://github.com/RomGai/VideoStir。
原文摘要 · Abstract (English)
Scaling multimodal large language models (MLLMs) to long videos is constrained by limited context windows. While retrieval-augmented generation (RAG) is a promising remedy by organizing query-relevant visual evidence into a compact context, most existing methods (i) flatten videos into independent segments, breaking their inherent spatio-temporal structure, and (ii) depend on explicit semantic matching, which can miss cues that are implicitly relevant to the query's intent. To overcome these limitations, we propose VideoStir, a structured and intent-aware long-video RAG framework. It firstly structures a video as a spatio-temporal graph at clip level, and then performs multi-hop retrieval to aggregate evidence across distant yet contextually related events. Furthermore, it introduces an MLLM-backed intent-relevance scorer that retrieves frames based on their alignment with the query's reasoning intent. To support this capability, we curate IR-600K, a large-scale dataset tailored for learning frame-query intent alignment. Experiments show that VideoStir is competitive with state-of-the-art baselines without relying on auxiliary information, highlighting the promise of shifting long-video RAG from flattened semantic matching to structured, intent-aware reasoning. Codes and checkpoints are available at https://github.com/RomGai/VideoStir.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。