用运动流引导对齐,解决事件相机与摄像头的时序错位问题。
Learning Flow-Guided Registration for RGB-Event Semantic Segmentation
- 通过光流和事件时间特征实现跨模态像素级配准
- 在四个数据集上显著提升分割精度,最高提升6.2% mIoU
- 适合做事件相机与视觉融合的算法研究者
事件相机捕捉微秒级运动信息,可补充RGB传感器的不足。但现有将RGB-事件感知视为融合问题的方法忽视了固有的时空与模态错位问题,不同于其他RGB-X感知场景。为此,本文将RGB-事件分割从融合重构为注册问题。提出BRENet:一种基于运动流引导的双向框架,自适应匹配异构模态间的对应关系。具体而言,利用时间对齐的光流作为粗粒度引导,并结合细粒度事件时间特征,生成精确的前向与后向像素配对,将固有运动滞后转化为由光流估计误差决定的项,弥合模态差异。此外,提出运动增强事件张量(MET),将稀疏事件流转换为密集且时序一致的表示形式。在四个大规模数据集上的实验验证了该方法的有效性,确立了流引导注册在RGB-事件分割中的前景。代码已开源。
原文摘要 · Abstract (English)
Event cameras capture microsecond-level motion cues that complement RGB sensors. However, the prevailing paradigm of treating RGB-Event perception as a fusion problem is ill-posed, as it ignores the intrinsic (i) Spatiotemporal and (ii) Modal Misalignment, unlike other RGB-X sensing domains. To tackle these limitations, we recast RGB-Event segmentation from fusion to registration. We propose BRENet, a novel flow-guided bidirectional framework that adaptively matches correspondence between the asymmetric modalities. Specifically, it leverages temporally aligned optical flows as a coarse-grained guide, along with fine-grained event temporal features, to generate precise forward and backward pixel pairings for registration. This pairing mechanism converts the inherent motion lag into terms governed by flow estimation error, bridging modality gaps. Moreover, we introduce Motion-Enhanced Event Tensor (MET), a new representation that transforms sparse event streams into a dense, temporally coherent form. Extensive experiments on four large-scale datasets validate our approach, establishing flow-guided registration as a promising direction for RGB-Event segmentation. Our code is available at: https://github.com/zyaocoder/BRENet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。