arXiv:2608.04426cs.CVcs.CL2026-08

模型看视频片段后,找其他视频中物体未来状态的匹配画面。

Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes

论文配图:Predict, Then Retrieve: Cross-Instance Future-State Retrieval from Video Prefixes
图 1 · 摘自论文原文
  • 用冻结编码器预测未来潜在表示,跨语义与视觉空间匹配。
  • 大模型仍远低于理想上限,说明预测才是核心难点。
  • 适合研究视频理解、跨实例检索与未来状态建模的学者。

我们提出预测性状态检索(PSR)任务:模型观察一段短视频和关于某物体未来状态的时间性问题,从其他视频或图像中检索出描绘该状态的实例。不同于动作预测、事件定位或视频生成,PSR结合了未来状态预测与跨实例检索,涵盖多个时间跨度。我们基于四个数据集构建了基准,包含分级的人工验证真值、难度层级及一个理想上限(oracle)。我们提出LFTR——一种轻量级检索器,使用冻结编码器,根据问题和时间跨度预测未来潜在表示,并在互补的语义与视觉空间中进行匹配。上限分解表明:一旦未来状态被指定,其可检索性极高;而所有评估过的预测器(包括具备前帧访问权限的大规模多模态语言模型)均远低于此上限。因此,预测而非感知是核心学习挑战。LFTR以显著更低的推理成本缩小了差距,消融实验显示其优势源于跨空间融合与难负样本训练,而非潜在表示滚动。我们已开源基准、代码与评估脚本。

原文摘要 · Abstract (English)

We introduce Predictive State Retrieval (PSR), a task in which a model observes a short video prefix and a temporal question about an object's future state, then retrieves instances from other videos or images that depict that state. Unlike action anticipation, which predicts a label, moment retrieval, which localizes an observed event within a video, or video generation, which synthesizes pixels, PSR combines anticipation with cross-instance retrieval across multiple temporal horizons. We construct a benchmark from four datasets with graded, human-validated ground truth, difficulty tiers, and an oracle ceiling. We also propose LFTR, a lightweight retriever with frozen encoders that predicts a question- and horizon-conditioned future latent and matches it in complementary semantic and visual spaces. A ceiling decomposition reveals a clear bottleneck: the true future state is highly retrievable once specified, whereas every predictor we evaluate, including a large multimodal language model with access to the prefix frames, remains far below the oracle. Thus, forecasting rather than perception is the central learnable challenge. LFTR narrows this gap at substantially lower inference cost, and ablations attribute its gains to cross-space fusion and hard-negative training rather than latent rollout. We release the benchmark, code, and evaluation scripts.

视频理解未来状态跨实例检索轻量模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。