用时间引导的视觉模型，让事件相机在复杂环境更准地识物。
Temporal-Guided Visual Foundation Models for Event-Based Vision
- 设计时间融合模块，捕捉长期时序依赖和多尺度时空关联。
- 在分割、深度估计、检测任务上分别提升16%、21%、16%，达新纪录。
- 适合做事件相机视觉任务的研究者与工业应用开发者。
事件相机在恶劣环境下具备独特优势，但处理异步事件流仍是难题。现有方法依赖专用架构或高资源训练，而基于图像预训练的视觉基础模型（VFMs）在事件视觉中的潜力尚未被充分挖掘。为此，我们提出时间引导的视觉基础模型（TGVFM），通过引入时序上下文融合模块，无缝集成预训练的VFM。该模块包含三个核心组件：(1) 长程时序注意力，建模全局时序依赖；(2) 双重时空注意力，实现多尺度帧关联；(3) 深层特征引导机制，融合语义-时序特征。通过在真实世界数据上微调事件到视频模型，并结合基于Transformer的VFM，TGVFM在保留时空动态的同时利用预训练表征。实验表明，在语义分割、深度估计、目标检测任务上分别相较现有方法提升16%、21%、16%，达到当前最优性能。本工作揭示了图像基视觉基础模型在事件视觉中跨模态推理的潜力。代码已开源：https://github.com/XiaRho/TGVFM。
原文摘要 · Abstract (English)
Event cameras offer unique advantages for vision tasks in challenging environments, yet processing asynchronous event streams remains an open challenge. While existing methods rely on specialized architectures or resource-intensive training, the potential of leveraging modern Visual Foundation Models (VFMs) pretrained on image data remains under-explored for event-based vision. To address this, we propose Temporal-Guided VFM (TGVFM), a novel framework that integrates VFMs with our temporal context fusion block seamlessly to bridge this gap. Our temporal block introduces three key components: (1) Long-Range Temporal Attention to model global temporal dependencies, (2) Dual Spatiotemporal Attention for multi-scale frame correlation, and (3) Deep Feature Guidance Mechanism to fuse semantic-temporal features. By retraining event-to-video models on real-world data and leveraging transformer-based VFMs, TGVFM preserves spatiotemporal dynamics while harnessing pretrained representations. Experiments demonstrate SoTA performance across semantic segmentation, depth estimation, and object detection, with improvements of 16%, 21%, and 16% over existing methods, respectively. Overall, this work unlocks the cross-modality potential of image-based VFMs for event-based vision with temporal reasoning. Code is available at https://github.com/XiaRho/TGVFM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。