用预训练视觉先验实现零样本大位移光流估计
MegaFlow: Zero-Shot Large Displacement Optical Flow
- 将光流估计转为全局匹配问题,利用ViT特征捕捉大位移
- 仅需轻量迭代优化,即在多个基准上达顶尖零样本性能
- 适合需要泛化运动估计的跨场景应用,如视频分析
大位移光流准确估计仍是关键挑战。现有方法通常依赖迭代局部搜索或领域特定微调,严重限制了其在大位移和零样本泛化场景下的表现。为此,我们提出MegaFlow,一种简单而强大的零样本大位移光流模型。不同于复杂任务定制的架构设计,MegaFlow通过适配强大的预训练视觉先验生成时序一致的运动场。具体地,我们利用预训练的全局Vision Transformer特征,将光流估计建模为全局匹配问题,自然捕获大位移;随后进行少量轻量级迭代优化,进一步提升亚像素精度。大量实验表明,MegaFlow在多个光流基准上实现顶尖的零样本性能。此外,该模型在长距离点跟踪基准上也表现出色,证明其强泛化能力,暗示了一种通用化的运动估计范式。项目主页:https://kristen-z.github.io/projects/megaflow。
原文摘要 · Abstract (English)
Accurate estimation of large displacement optical flow remains a critical challenge. Existing methods typically rely on iterative local search or/and domain-specific fine-tuning, which severely limits their performance in large displacement and zero-shot generalization scenarios. To overcome this, we introduce MegaFlow, a simple yet powerful model for zero-shot large displacement optical flow. Rather than relying on highly complex, task-specific architectural designs, MegaFlow adapts powerful pre-trained vision priors to produce temporally consistent motion fields. In particular, we formulate flow estimation as a global matching problem by leveraging pre-trained global Vision Transformer features, which naturally capture large displacements. This is followed by a few lightweight iterative refinements to further improve the sub-pixel accuracy. Extensive experiments demonstrate that MegaFlow achieves state-of-the-art zero-shot performance across multiple optical flow benchmarks. Moreover, our model also delivers highly competitive zero-shot performance on long-range point tracking benchmarks, demonstrating its robust transferability and suggesting a unified paradigm for generalizable motion estimation. Our project page is at: https://kristen-z.github.io/projects/megaflow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。