让视频大模型学会物理运动规律,提升对动态世界的理解能力。
PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglement
- 分离外观与运动信号,用双分支编码器实现解耦
- 引入神经微分方程建模连续物理动态,生成可微表示
- 自监督学习避免标注依赖,适合需要物理推理的场景
视频大语言模型在多种视频-语言任务中表现优异,但在需深入理解物理动态的场景下表现不佳,主要因依赖外观匹配。为突破此限制,本文提出PhyVLLM,一种基于物理引导的视频-语言框架。该模型通过双分支编码器显式解耦视觉外观与物体运动。为建模时间上的物理动态,引入神经常微分方程(Neural ODE)模块,生成可微的物理动态表示。运动感知特征被映射至预训练大语言模型的词元空间,实现物理推理而不损害原有多模态能力。为规避物理属性标注成本,采用自监督方式建模物体运动的连续演化。实验表明,PhyVLLM在物理推理和通用视频理解任务上均显著优于现有先进模型,验证了显式物理建模的有效性。
原文摘要 · Abstract (English)
Video Large Language Models (Video LLMs) have shown impressive performance across a wide range of video-language tasks. However, they often fail in scenarios requiring a deeper understanding of physical dynamics. This limitation primarily arises from their reliance on appearance-based matching. Incorporating physical motion modeling is crucial for deeper video understanding, but presents three key challenges: (1) motion signals are often entangled with appearance variations, making it difficult to extract clean physical cues; (2) effective motion modeling requires not only continuous-time motion representations but also capturing physical dynamics; and (3) collecting accurate annotations for physical attributes is costly and often impractical. To address these issues, we propose PhyVLLM, a physical-guided video-language framework that explicitly incorporates physical motion into Video LLMs. Specifically, PhyVLLM disentangles visual appearance and object motion through a dual-branch encoder. To model physical dynamics over time, we incorporate a Neural Ordinary Differential Equation (Neural ODE) module, which generates differentiable physical dynamic representations. The resulting motion-aware representations are projected into the token space of a pretrained LLM, enabling physics reasoning without compromising the model's original multimodal capabilities. To circumvent the need for explicit physical labels, PhyVLLM employs a self-supervised manner to model the continuous evolution of object motion. Experimental results demonstrate that PhyVLLM significantly outperforms state-of-the-art Video LLMs on both physical reasoning and general video understanding tasks, highlighting the advantages of incorporating explicit physical modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。