用扩散模型先验实现单图运动估计,1步完成且快100倍以上。
StableMotion: One-Step Motion Estimation with Diffusion Prior
- 将扩散模型转为图像到运动的估计器,利用几何与内容先验。
- 提出自适应集成策略,提升输出一致性与保真度。
- 发现采样步数越多越差,支持1步推理,效率大幅提升。
我们提出StableMotion,一种新颖框架,利用预训练大规模图像扩散模型中的几何与内容先验,解决单图校正任务中的运动估计问题,如拼接图像校正(SIR)和滚动快门校正(RSC)。StableMotion以文本到图像的Stable Diffusion(SD)模型为骨干,将其重用于图像到运动的估计。为缓解扩散模型产生的不一致输出,我们提出自适应集成策略(AES),将多次输出融合为一致、高保真的结果。此外,我们发现了采样步数灾难(SSD)这一反直觉现象:增加采样步数反而导致性能下降,这促使我们采用1步推理设计。StableMotion在两项图像校正任务中均达到最先进水平,并通过定性示例和无参考评估展示了在未见数据集(SIR-OOD)和真实拍摄数据(RSC)上的良好迁移能力。得益于SSD发现,StableMotion实现高效1步推理,相比此前基于扩散模型的方法提速超100倍,即使结合可选的AES后处理亦然。代码与权重已公开于https://github.com/ivowang/StableMotion。
原文摘要 · Abstract (English)
We present StableMotion, a novel framework that leverages geometric and content priors from pretrained large-scale image diffusion models for motion estimation in single-image rectification tasks such as Stitched Image Rectangling (SIR) and Rolling Shutter Correction (RSC). Specifically, StableMotion takes a text-to-image Stable Diffusion (SD) model as its backbone and repurposes it as an image-to-motion estimator. To mitigate inconsistent outputs produced by diffusion models, we propose Adaptive Ensemble Strategy (AES), which consolidates multiple outputs into a cohesive, high-fidelity result. Additionally, we present Sampling Steps Disaster (SSD), a counterintuitive phenomenon in which increasing the number of sampling steps can lead to poorer outcomes, motivating our one-step inference design. StableMotion is evaluated on two image rectification tasks and delivers state-of-the-art performance on both, while also showing promising transferability through qualitative examples and no-reference evaluations on unseen SIR-OOD and real-captured RSC benchmarks. Supported by SSD, StableMotion achieves efficient one-step inference, offering over 100$\times$ speedup compared to previous diffusion model-based methods even when combined with the optional AES post-processing. Code and weights are available at https://github.com/ivowang/StableMotion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。