arXiv:2507.09082cs.CV2025-07NeurIPS被引 4

无需微调,用生成模型零样本提取高质量光流

Taming generative video models for zero-shot optical flow extraction

  • 通过注入微小扰动并追踪其传播,实现零样本光流提取
  • 在真实数据集TAP-Vid DAVIS上达到顶尖水平,优于传统方法
  • 适用于无标注数据场景,适合追求高效推理的研究者

从视频中提取光流仍是计算机视觉的核心问题。受大型通用模型成功的启发,我们探索是否可直接提示仅以预测未来帧为目标的冻结自监督视频模型,无需微调即可输出光流。以往尝试从视频生成器中读取深度或光照需微调,而光流缺乏标注数据且合成数据存在仿真到现实的差距。受反事实世界模型(CWM)启发,我们将其扩展至生成式视频模型,实现零样本光流提取。研究发现,成功的关键在于三个模型特性:(1) 未来帧的分布式预测(避免模糊或噪声输出);(2) 因子化潜在变量,独立处理每个时空块;(3) 随机访问解码,可基于任意未来像素子集进行条件生成。这些特性仅在新提出的局部随机访问序列(LRAS)架构中具备。基于此,我们提出KL追踪:一种测试时推理方法,将局部扰动注入首帧,滚动模型一步,计算扰动与未扰动预测分布间的KL散度。无需任何光流专用微调,该方法在真实世界TAP-Vid DAVIS和合成数据集TAP-Vid Kubric上表现媲美最先进任务特定模型。结果表明,对可控生成视频模型进行反事实提示,是高质量光流提取的有效替代方案。

原文摘要 · Abstract (English)

Extracting optical flow from videos remains a core computer vision problem. Motivated by the recent success of large general-purpose models, we ask whether frozen self-supervised video models trained only to predict future frames can be prompted, without fine-tuning, to output flow. Prior attempts to read out depth or illumination from video generators required fine-tuning; that strategy is ill-suited for flow, where labeled data is scarce and synthetic datasets suffer from a sim-to-real gap. Inspired by the Counterfactual World Model (CWM) paradigm, which can obtain point-wise correspondences by injecting a small tracer perturbation into a next-frame predictor and tracking its propagation, we extend this idea to generative video models for zero-shot flow extraction. We explore several popular architectures and find that successful zero-shot flow extraction in this manner is aided by three model properties: (1) distributional prediction of future frames (avoiding blurry or noisy outputs); (2) factorized latents that treat each spatio-temporal patch independently; and (3) random-access decoding that can condition on any subset of future pixels. These properties are uniquely present in the recently introduced Local Random Access Sequence (LRAS) architecture. Building on LRAS, we propose KL-tracing: a novel test-time inference procedure that injects a localized perturbation into the first frame, rolls out the model one step, and computes the Kullback-Leibler divergence between perturbed and unperturbed predictive distributions. Without any flow-specific fine-tuning, our method is competitive with state-of-the-art, task-specific models on the real-world TAP-Vid DAVIS benchmark and the synthetic TAP-Vid Kubric. Our results show that counterfactual prompting of controllable generative video models is an effective alternative to supervised or photometric-loss methods for high-quality flow.

光流估计生成模型零样本扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。