构建物理不可信视频检测数据集,评估文本生成视频模型的物理合理性。
PhyDetEx: Detecting and Explaining the Physical Plausibility of T2V Models
- 用改写真实视频描述的方式生成2588对对比视频,构建物理不可信检测数据集。
- 微调视觉语言模型,使其能识别并解释视频中违反物理规律的事件。
- 发现当前主流生成模型仍难保证物理合理性,尤其开源模型表现更弱。
随着模型容量与训练规模的提升,文本到视频(T2V)生成模型在视频质量、长度和指令遵循能力方面取得显著进展。然而,这些模型是否理解物理规律并生成符合物理常识的视频仍是未知。尽管视觉语言模型(VLMs)被广泛用于各类评估任务,但其难以识别生成视频中的物理不可能内容。为此,我们构建了名为 extbf{PID}(物理不可信检测)的数据集,包含500个手动标注的测试视频和2,588对成对训练视频;每对中,不可信视频由精心改写真实视频的描述生成,诱导T2V模型产生物理不合理内容。基于该数据集,我们提出一种轻量级微调方法,使VLM不仅能检测物理不可信事件,还能生成关于违反物理原理的文本解释。以微调后的VLM作为物理合理性检测与解释器,即 extbf{PhyDetEx},我们对多个先进T2V模型进行基准测试,评估其遵守物理定律的程度。结果表明,尽管近期T2V模型在生成物理合理视频方面有明显进步,但理解并遵循物理法则仍是挑战性问题,尤其在开源模型中更为突出。相关数据集、代码与检查点已公开于 https://github.com/Zeqing-Wang/PhyDetEx。
原文摘要 · Abstract (English)
Driven by the growing capacity and training scale, Text-to-Video (T2V) generation models have recently achieved substantial progress in video quality, length, and instruction-following capability. However, whether these models can understand physics and generate physically plausible videos remains a question. While Vision-Language Models (VLMs) have been widely used as general-purpose evaluators in various applications, they struggle to identify the physically impossible content from generated videos. To investigate this issue, we construct a \textbf{PID} (\textbf{P}hysical \textbf{I}mplausibility \textbf{D}etection) dataset, which consists of a \textit{test split} of 500 manually annotated videos and a \textit{train split} of 2,588 paired videos, where each implausible video is generated by carefully rewriting the caption of its corresponding real-world video to induce T2V models producing physically implausible content. With the constructed dataset, we introduce a lightweight fine-tuning approach, enabling VLMs to not only detect physically implausible events but also generate textual explanations on the violated physical principles. Taking the fine-tuned VLM as a physical plausibility detector and explainer, namely \textbf{PhyDetEx}, we benchmark a series of state-of-the-art T2V models to assess their adherence to physical laws. Our findings show that although recent T2V models have made notable progress toward generating physically plausible content, understanding and adhering to physical laws remains a challenging issue, especially for open-source models. Our dataset, training code, and checkpoints are available at \href{https://github.com/Zeqing-Wang/PhyDetEx}{https://github.com/Zeqing-Wang/PhyDetEx}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。