将事件相机数据对齐到图像模型潜空间,实现零样本跨模态感知
REALM: An RGB- and Event-Aligned Latent Manifold for Cross-Modal Perception

- 通过低秩适配将事件流映射到预训练图像模型的潜空间
- 在深度估计与语义分割任务上达到顶尖性能,无需微调
- 支持直接调用冻结的图像解码器处理原始事件数据,适合多模态研究者
事件相机相较于传统帧式传感器具有高时间分辨率、低延迟和强光照鲁棒性等优势。然而,现有基于学习的事件处理方法通常局限于特定任务,缺乏跨模态泛化能力。为此,我们提出REALM,一种跨模态框架,通过将事件表示投影到预训练的RGB基础模型潜空间,构建RGB与事件对齐的潜在流形。不依赖任务特定训练,而是采用低秩适配(LoRA)弥合模态差距,有效利用冻结的RGB骨干网络的几何与语义先验来处理异步事件流。实验表明,REALM能成功将事件数据映射至ViT-based基础模型潜空间。通过简单迁移在RGB教师模型上训练的线性分类头,即可完成深度估计、语义分割等下游任务。最显著的是,REALM实现了复杂冻结图像解码器(如MASt3R)对原始事件数据的直接零样本应用。在宽基线特征匹配任务中表现优于专用架构,达到当前最优水平。代码与模型已公开于https://papers.starslab.ca/realm/。
原文摘要 · Abstract (English)
Event cameras provide several unique advantages over standard frame-based sensors, including high temporal resolution, low latency, and robustness to extreme lighting. However, existing learning-based approaches for event processing are typically confined to narrow, task-specific silos and lack the ability to generalize across modalities. We address this gap with REALM, a cross-modal framework that learns an RGB- and Event-Aligned Latent Manifold by projecting event representations into the pretrained latent space of RGB foundation models. Instead of task-specific training, we leverage low-rank adaptation (LoRA) to bridge the modality gap, effectively unlocking the geometric and semantic priors of frozen RGB backbones for asynchronous event streams. We demonstrate that REALM effectively maps events into the ViT-based foundation latent space. Our method performs downstream tasks, such as depth estimation and semantic segmentation, by simply transferring linear heads trained on the RGB teacher. Most significantly, REALM enables the direct, zero-shot application of complex, frozen image-trained decoders, such as MASt3R, to raw event data. We demonstrate state-of-the-art performance in wide-baseline feature matching, significantly outperforming specialized architectures. Code and models are available at https://papers.starslab.ca/realm/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。