从落叶视频中反推风场，让计算机看见看不见的物理力。
Seeing the Wind from a Falling Leaf
- 端到端可微分逆图形框架，联合建模物体几何、物理属性与作用力
- 通过反向传播从运动轨迹恢复出合理的风场分布
- 适合对物理规律建模、视频生成与编辑感兴趣的读者
计算机视觉长期目标是建模视频中的运动，但导致物体变形与运动的不可见物理交互仍鲜有研究。本文提出从视觉观测中恢复不可见力的方法，例如通过观察落叶下落推断风场。核心创新是一个端到端可微分的逆图形框架，能直接从视频中联合建模物体几何、物理属性及相互作用。借助反向传播，该方法可实现从运动轨迹恢复力场表示。我们在合成数据与真实场景上验证了方法的有效性，结果表明其能从视频中推断出合理且物理一致的力场。此外，展示了在基于物理的视频生成与编辑中的应用潜力。我们希望该方法能推动对像素背后物理过程的理解，弥合视觉与物理之间的鸿沟。更多视频结果请访问项目页。
原文摘要 · Abstract (English)
A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our \href{https://chaoren2357.github.io/seeingthewind/}{project page}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。