用图结构建模视频运动，更准更省显存。
Motion Graph Unleashed: A Novel Approach to Video Prediction
- 将视频块转为图节点，捕捉时空关系
- 在UCF Sports上模型缩小78%，显存降47%
- 适合需要轻量化视频预测的场景
我们提出运动图（motion graph），一种全新的视频预测方法，可从有限历史帧中预测未来帧。该方法将视频帧块转化为相互连接的图节点，全面描述其时空关系。相比图像差分、光流和运动矩阵等传统运动表示方式，运动图在捕捉复杂运动模式方面更具优势，且内存消耗更低。我们进一步构建了基于运动图的视频预测流程，在UCF Sports、KITTI和Cityscapes等多个数据集上验证了其强大表征能力。尤其在UCF Sports上，本方法性能达到当前最优水平，同时模型规模减少78%，GPU显存使用量降低47%。
原文摘要 · Abstract (English)
We introduce motion graph, a novel approach to the video prediction problem, which predicts future video frames from limited past data. The motion graph transforms patches of video frames into interconnected graph nodes, to comprehensively describe the spatial-temporal relationships among them. This representation overcomes the limitations of existing motion representations such as image differences, optical flow, and motion matrix that either fall short in capturing complex motion patterns or suffer from excessive memory consumption. We further present a video prediction pipeline empowered by motion graph, exhibiting substantial performance improvements and cost reductions. Experiments on various datasets, including UCF Sports, KITTI and Cityscapes, highlight the strong representative ability of motion graph. Especially on UCF Sports, our method matches and outperforms the SOTA methods with a significant reduction in model size by 78% and a substantial decrease in GPU memory utilization by 47%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。