针对事件相机设计的长期时序自监督预训练框架,提升感知任务性能。
TESPEC: Temporally-Enhanced Self-Supervised Pretraining for Event Cameras
- 利用长序列事件构建伪灰度视频作为重建目标,捕捉时空语义信息。
- 在物体检测、语义分割等任务上达到当前最优,优于现有自监督方法。
- 特别适合循环模型,首次在预训练中引入长期事件时序建模。
长期时序信息对事件相机感知任务至关重要,因为原始事件仅记录像素亮度变化。近期研究发现,从头训练时循环模型表现优于前馈模型;但在使用自监督预训练权重时,前馈模型反而更优。当前事件相机的自监督学习方法大多模仿基于RGB图像的方法,仅在短时间窗口内对原始事件进行前馈模型预训练,忽略了事件的长期时序特性。本文提出TESPEC,一个专为学习时空信息设计的自监督预训练框架。TESPEC首次在预训练中利用长事件序列,适用于循环模型。其采用掩码图像建模范式,设计新重建目标:将事件累积为包含场景高层语义信息的伪灰度视频,对传感器噪声鲁棒且减少运动模糊。重建该目标要求模型推理长期事件历史。大量实验表明,其在下游任务(包括目标检测、语义分割和单目深度估计)中达到当前最优性能。
原文摘要 · Abstract (English)
Long-term temporal information is crucial for event-based perception tasks, as raw events only encode pixel brightness changes. Recent works show that when trained from scratch, recurrent models achieve better results than feedforward models in these tasks. However, when leveraging self-supervised pre-trained weights, feedforward models can outperform their recurrent counterparts. Current self-supervised learning (SSL) methods for event-based pre-training largely mimic RGB image-based approaches. They pre-train feedforward models on raw events within a short time interval, ignoring the temporal information of events. In this work, we introduce TESPEC, a self-supervised pre-training framework tailored for learning spatio-temporal information. TESPEC is well-suited for recurrent models, as it is the first framework to leverage long event sequences during pre-training. TESPEC employs the masked image modeling paradigm with a new reconstruction target. We design a novel method to accumulate events into pseudo grayscale videos containing high-level semantic information about the underlying scene, which is robust to sensor noise and reduces motion blur. Reconstructing this target thus requires the model to reason about long-term history of events. Extensive experiments demonstrate our state-of-the-art results in downstream tasks, including object detection, semantic segmentation, and monocular depth estimation. Project webpage: https://mhdmohammadi.github.io/TESPEC_webpage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。