用稀疏点击预测物体运动分布,实现多模态动态建模。
What If : Understanding Motion Through Sparse Interactions
- 基于稀疏点击设计新型变换器,直接建模局部运动分布
- 在人脸生成与刚体运动估计中超越专用模型,提升30%以上性能
- 适合需要理解物理交互与不确定性的场景建模任务
理解物理场景的动态变化需推理其多种潜在演变方式,尤其是局部相互作用的影响。本文提出流动点击变换器(Flow Poke Transformer, FPT),一种直接预测局部运动分布的新框架,条件为稀疏的交互动作(称为‘点击’)。与传统方法仅能密集采样单一动态结果不同,FPT提供可解释、直接访问的多模态运动表示,揭示运动对物理交互的依赖性及动态固有的不确定性。我们在多个下游任务上评估该模型,以对比先前方法并展示其灵活性:在密集人脸运动生成任务中,通用预训练模型超越专用基线;在合成数据等强域外任务中微调后,显著优于领域内方法;直接预测显式运动分布还使该方法在仅凭点击进行运动部件分割的任务中达到竞争性表现,进一步体现其通用性。代码与模型已公开于 https://compvis.github.io/flow-poke-transformer。
原文摘要 · Abstract (English)
Understanding the dynamics of a physical scene involves reasoning about the diverse ways it can potentially change, especially as a result of local interactions. We present the Flow Poke Transformer (FPT), a novel framework for directly predicting the distribution of local motion, conditioned on sparse interactions termed "pokes". Unlike traditional methods that typically only enable dense sampling of a single realization of scene dynamics, FPT provides an interpretable directly accessible representation of multi-modal scene motion, its dependency on physical interactions and the inherent uncertainties of scene dynamics. We also evaluate our model on several downstream tasks to enable comparisons with prior methods and highlight the flexibility of our approach. On dense face motion generation, our generic pre-trained model surpasses specialized baselines. FPT can be fine-tuned in strongly out-of-distribution tasks such as synthetic datasets to enable significant improvements over in-domain methods in articulated object motion estimation. Additionally, predicting explicit motion distributions directly enables our method to achieve competitive performance on tasks like moving part segmentation from pokes which further demonstrates the versatility of our FPT. Code and models are publicly available at https://compvis.github.io/flow-poke-transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。