让视频生成模型的运动理解可解释，精准定位何时何地谁在动。
Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers
- 提出GramCol方法，为文本概念生成逐帧显著图。
- 无需梯度计算即可实现运动时空精确定位，零样本分割效果好。
- 适用于运动与非运动概念，适合研究生成机制的学者。
视频扩散变换器（Video DiTs）能根据文本描述生成高质量、高保真视频，但其如何将运动词汇转化为视觉动作仍不清晰。现有可解释性研究多聚焦物体，对视频中的运动行为关注不足。本文旨在明确特定运动概念对应的时间与空间位置。首先，提出GramCol方法，自适应生成任意文本概念（包括运动与非运动）的逐帧显著图；其次，设计运动特征选择算法，构建可解释的运动注意力图（IMAP），实现运动在时空上的双重定位。该方法无需梯度计算或参数更新，实验表明其在运动定位任务和零样本视频语义分割上表现优异，能为运动与非运动概念提供清晰、可解释的显著图。
原文摘要 · Abstract (English)
Video Diffusion Transformers (DiTs) have been synthesizing high-quality video with high fidelity from given text descriptions involving motion. However, understanding how Video DiTs convert motion words into video remains insufficient. Furthermore, while prior studies on interpretable saliency maps primarily target objects, motion-related behavior in Video DiTs remains largely unexplored. In this paper, we investigate concrete motion features that specify when and which object moves for a given motion concept. First, to spatially localize, we introduce GramCol, which adaptively produces per-frame saliency maps for any text concept, including both motion and non-motion. Second, we propose a motion-feature selection algorithm to obtain an Interpretable Motion-Attentive Map (IMAP) that localizes motion spatially and temporally. Our method discovers concept saliency maps without the need for any gradient calculation or parameter update. Experimentally, our method shows outstanding localization capability on the motion localization task and zero-shot video semantic segmentation, providing interpretable and clearer saliency maps for both motion and non-motion concepts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。