用压缩视频的运动矢量评估生成视频的时序真实感。
Temporal Realism Evaluation of Generated Videos Using Compressed-Domain Motion Vectors
- 从压缩流提取运动矢量，量化真实与生成视频的运动差异。
- 发现生成视频普遍存在中心偏移、块状运动等时序缺陷。
- 融合运动矢量可显著提升判别模型对生成视频的识别能力。
当前生成视频模型在时序真实性上仍存在明显短板，多数评估指标侧重空间外观而对运动不敏感。本文提出一种可扩展、模型无关的框架，直接从压缩视频流中提取编码器生成的运动矢量（MVs），利用H.264和HEVC标准提供的轻量级、分辨率一致的运动动态描述。通过计算真实与生成视频运动矢量统计间的Kullback-Leibler、Jensen-Shannon和Wasserstein散度来量化真实性。在包含8个顶尖生成器的GenVidBench数据集上的实验显示：熵相关散度表明Pika和SVD最接近真实视频，MV总和统计则偏好VC2和Text2Video-Zero，CogVideo在两项指标上均表现最差。运动场可视化与类别条件运动热图揭示了中心偏置、稀疏分段恒定流及网格状伪影，这些是帧级指标无法捕捉的。此外，通过通道拼接、交叉注意力、联合嵌入和运动感知融合模块研究了MV-RGB融合，结果表明引入运动矢量后，基于ResNet、I3D和TSN的下游分类任务性能显著提升，其中ResNet-18和ResNet-34准确率最高达97.4%，I3D达到99.0%的真实/生成判别准确率。结果表明，压缩域运动矢量能有效诊断生成视频的运动缺陷，并增强判别模型的时序推理能力。代码已开源：https://github.com/KurbanIntelligenceLab/Motion-Vector-Learning
原文摘要 · Abstract (English)
Temporal realism remains a central weakness of current generative video models, as most evaluation metrics prioritize spatial appearance and offer limited sensitivity to motion. We introduce a scalable, model-agnostic framework that assesses temporal behavior using motion vectors (MVs) extracted directly from compressed video streams. Codec-generated MVs from standards such as H.264 and HEVC provide lightweight, resolution-consistent descriptors of motion dynamics. We quantify realism by computing Kullback-Leibler, Jensen-Shannon, and Wasserstein divergences between MV statistics of real and generated videos. Experiments on the GenVidBench dataset containing videos from eight state-of-the-art generators reveal systematic discrepancies from real motion: entropy-based divergences rank Pika and SVD as closest to real videos, MV-sum statistics favor VC2 and Text2Video-Zero, and CogVideo shows the largest deviations across both measures. Visualizations of MV fields and class-conditional motion heatmaps further reveal center bias, sparse and piecewise constant flows, and grid-like artifacts that frame-level metrics do not capture. Beyond evaluation, we investigate MV-RGB fusion through channel concatenation, cross-attention, joint embedding, and a motion-aware fusion module. Incorporating MVs improves downstream classification across ResNet, I3D, and TSN backbones, with ResNet-18 and ResNet-34 reaching up to 97.4% accuracy and I3D achieving 99.0% accuracy on real-versus-generated discrimination. These findings demonstrate that compressed-domain MVs provide an effective temporal signal for diagnosing motion defects in generative videos and for strengthening temporal reasoning in discriminative models. The implementation is available at: https://github.com/KurbanIntelligenceLab/Motion-Vector-Learning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。