一个统一模型同时处理视频低层感知任务，速度快且效果好。
L4P: Towards Unified Low-Level 4D Vision Perception
- 用预训练ViT视频编码器+轻量任务头，统一建模视频时空关系。
- 在深度、光流等密集任务和追踪等稀疏任务上表现媲美专用模型。
- 单次推理可完成所有任务，耗时接近单任务方法。
视频像素间的时空关系蕴含低层4D感知任务的关键信息。单一模型若能有效建模此关系，应可通用解决多种任务。然而，当前多数先进方法依赖针对特定任务设计的架构。本文提出L4P，一种前馈式、通用型架构，在统一框架下解决低层4D感知任务。L4P采用预训练的基于ViT的视频编码器，并搭配轻量级任务专用头部，无需大量微调。尽管是通用前馈结构，其在密集任务（如深度估计、光流估计）和稀疏任务（如2D/3D追踪）上均达到与现有专用方法相当的性能。更重要的是，它能在与单任务方法相近的时间内一次性完成所有任务。
原文摘要 · Abstract (English)
The spatio-temporal relationship between the pixels of a video carries critical information for low-level 4D perception tasks. A single model that reasons about it should be able to solve several such tasks well. Yet, most state-of-the-art methods rely on architectures specialized for the task at hand. We present L4P, a feedforward, general-purpose architecture that solves low-level 4D perception tasks in a unified framework. L4P leverages a pre-trained ViT-based video encoder and combines it with per-task heads that are lightweight and therefore do not require extensive training. Despite its general and feedforward formulation, our method is competitive with existing specialized methods on both dense tasks, such as depth or optical flow estimation, and sparse tasks, such as 2D/3D tracking. Moreover, it solves all tasks at once in a time comparable to that of single-task methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。