用点轨迹提升视频预测的时序一致性与效率
Tracktention: Leveraging Point Tracking to Attend Videos Faster and Better
- 引入点轨迹作为运动信息，增强视频帧间对齐
- 在深度与彩色化任务中显著改善时序一致性
- 可无缝嵌入现有模型，升级图像模型为视频领先模型
时序一致性对视频预测至关重要，以确保输出连贯且无伪影。传统方法如时间注意力和3D卷积在大幅物体运动或动态场景中的长时依赖建模上表现不佳。为此，我们提出Tracktention Layer，一种新型架构组件，通过点轨迹（即跨帧对应点序列）显式整合运动信息。该设计增强了时序对齐，有效处理复杂运动，保持特征表示的时序一致性。本方法计算高效，可无缝集成至现有模型（如Vision Transformers），仅需少量修改即可将图像模型升级为当前最优视频模型，在视频深度预测和视频着色任务中均优于基线，性能甚至超过专为视频设计的模型。
原文摘要 · Abstract (English)
Temporal consistency is critical in video prediction to ensure that outputs are coherent and free of artifacts. Traditional methods, such as temporal attention and 3D convolution, may struggle with significant object motion and may not capture long-range temporal dependencies in dynamic scenes. To address this gap, we propose the Tracktention Layer, a novel architectural component that explicitly integrates motion information using point tracks, i.e., sequences of corresponding points across frames. By incorporating these motion cues, the Tracktention Layer enhances temporal alignment and effectively handles complex object motions, maintaining consistent feature representations over time. Our approach is computationally efficient and can be seamlessly integrated into existing models, such as Vision Transformers, with minimal modification. It can be used to upgrade image-only models to state-of-the-art video ones, sometimes outperforming models natively designed for video prediction. We demonstrate this on video depth prediction and video colorization, where models augmented with the Tracktention Layer exhibit significantly improved temporal consistency compared to baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。