用数字孪生视频表示+大模型推理,让视频检索能理解隐含语义。
Reasoning Text-to-Video Retrieval via Digital Twin Video Representations and Large Language Models
- 将视频拆解为数字孪生结构,支持大模型直接推理长时序内容
- 在135视频数据集上达到81.2% R@1,领先基线超50个百分点
- 适合需要理解复杂语义、带逻辑推理的视频搜索任务
文本到视频检索的目标是根据文本查询从大规模数据库中查找相关视频。现有方法已能处理显式查询(即视觉内容被明确描述),但在处理隐式查询(需通过推理才能识别相关视频)时表现不佳。本文提出「推理型文本到视频检索」,通过推理能力处理隐式查询,并提供对象级定位掩码以标出满足条件的实体。不同于直接依赖视觉-语言模型,我们采用数字孪生视频表示——即通过专用视觉模型分解显著物体的结构化场景表示。该方法使大语言模型能直接对长时序视频内容进行推理,无需视觉标记压缩。我们的两阶段框架首先对分解后的子查询与数字孪生表示进行组合对齐以候选视频,再通过大语言模型进行就地精炼推理,调用额外专用模型填补信息空白。我们构建了两个基准:447个手工创建的隐式查询对应135个视频(ReasonT2VBench-135),以及更具挑战性的1000视频版本(ReasonT2VBench-1000)。所提方法在ReasonT2VBench-135上取得81.2% R@1,优于最强基线超过50个百分点,并在扩展配置下保持81.7% R@1,同时在三个传统基准(MSR-VTT、MSVD、VATEX)上达到最新水平。
原文摘要 · Abstract (English)
The goal of text-to-video retrieval is to search large databases for relevant videos based on text queries. Existing methods have progressed to handling explicit queries where the visual content of interest is described explicitly; however, they fail with implicit queries where identifying videos relevant to the query requires reasoning. We introduce reasoning text-to-video retrieval, a paradigm that extends traditional retrieval to process implicit queries through reasoning while providing object-level grounding masks that identify which entities satisfy the query conditions. Instead of relying on vision-language models directly, we propose representing video content as digital twins, i.e., structured scene representations that decompose salient objects through specialist vision models. This approach is beneficial because it enables large language models to reason directly over long-horizon video content without visual token compression. Specifically, our two-stage framework first performs compositional alignment between decomposed sub-queries and digital twin representations for candidate identification, then applies large language model-based reasoning with just-in-time refinement that invokes additional specialist models to address information gaps. We construct a benchmark of 447 manually created implicit queries with 135 videos (ReasonT2VBench-135) and another more challenging version of 1000 videos (ReasonT2VBench-1000). Our method achieves 81.2% R@1 on ReasonT2VBench-135, outperforming the strongest baseline by greater than 50 percentage points, and maintains 81.7% R@1 on the extended configuration while establishing state-of-the-art results in three conventional benchmarks (MSR-VTT, MSVD, and VATEX).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。