FRAME通过预测未来特征，实现视频帧的高精度时空一致表示。
FRAME: Pre-Training Video Feature Representations via Anticipation and Memory
- 用过去和当前图像预测DINO块特征，建模时序一致性。
- 在7个数据集6项密集任务上超越图像编码器和现有视频模型。
- 兼顾精度与轻量，适合需要细粒度视觉对应的任务。
密集视频预测任务（如目标跟踪、语义分割）需要为每帧生成时空一致且空间密集的特征。然而现有方法存在不足：图像编码器（如DINO或CLIP）缺乏时序感知，而视频模型（如VideoMAE）在密集预测任务上的表现不如图像编码器。为此，我们提出FRAME，一种面向密集视频理解的自监督视频帧编码器。FRAME通过从过去的和当前的RGB帧中预测当前及未来的DINO块特征，生成空间精确且时序连贯的表示。据我们所知，FRAME是首个利用基于图像的模型实现密集预测并在此类任务上超越它们的视频编码器。作为辅助能力，FRAME将类别标记对齐至CLIP的语义空间，支持语言驱动的任务（如视频分类）。我们在七个数据集上评估了六个密集预测任务，结果表明其始终优于图像编码器和现有的自监督视频模型。尽管功能多样，FRAME仍保持紧凑架构，适用于多种下游应用。
原文摘要 · Abstract (English)
Dense video prediction tasks, such as object tracking and semantic segmentation, require video encoders that generate temporally consistent, spatially dense features for every frame. However, existing approaches fall short: image encoders like DINO or CLIP lack temporal awareness, while video models such as VideoMAE underperform compared to image encoders on dense prediction tasks. We address this gap with FRAME, a self-supervised video frame encoder tailored for dense video understanding. FRAME learns to predict current and future DINO patch features from past and present RGB frames, leading to spatially precise and temporally coherent representations. To our knowledge, FRAME is the first video encoder to leverage image-based models for dense prediction while outperforming them on tasks requiring fine-grained visual correspondence. As an auxiliary capability, FRAME aligns its class token with CLIP's semantic space, supporting language-driven tasks such as video classification. We evaluate FRAME across six dense prediction tasks on seven datasets, where it consistently outperforms image encoders and existing self-supervised video models. Despite its versatility, FRAME maintains a compact architecture suitable for a range of downstream applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。