用预训练视觉模型的语义特征预测未来，高效且通用。
DINO-Foresight: Looking into the Future with DINO
- 在预训练视觉模型的语义特征空间中进行自监督预测
- 在多个任务上表现优异，具备强鲁棒性与可扩展性
- 适合自动驾驶、机器人等需未来感知的场景
预测未来动态对自动驾驶和机器人等应用至关重要。现有像素级方法计算开销大，常关注无关细节。为此，我们提出DINO-Foresight，一种在预训练视觉基础模型（VFMs）语义特征空间中运行的新框架。该方法通过自监督方式训练一个掩码特征变换器，以预测VFM特征随时间的演变。通过预测这些特征，可直接使用现成的任务特定头完成各类场景理解任务。在此框架中，VFM特征被视为潜在空间，不同任务头可附加其上实现未来帧分析。大量实验表明，本框架在性能、鲁棒性和可扩展性方面均表现卓越。
原文摘要 · Abstract (English)
Predicting future dynamics is crucial for applications like autonomous driving and robotics, where understanding the environment is key. Existing pixel-level methods are computationally expensive and often focus on irrelevant details. To address these challenges, we introduce DINO-Foresight, a novel framework that operates in the semantic feature space of pretrained Vision Foundation Models (VFMs). Our approach trains a masked feature transformer in a self-supervised manner to predict the evolution of VFM features over time. By forecasting these features, we can apply off-the-shelf, task-specific heads for various scene understanding tasks. In this framework, VFM features are treated as a latent space, to which different heads attach to perform specific tasks for future-frame analysis. Extensive experiments show the very strong performance, robustness and scalability of our framework. Project page and code at https://dino-foresight.github.io/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。