无需标签和硬件对齐,实现事件图像跨模态匹配新方法
Label-Free Target-Domain Adaptation for Unconstrained Event-Image Feature Matching via Dual-Stage Distillation

- 两阶段训练:先无标签预训练,再自蒸馏适应目标域
- 在MVSEC和TUM-VIE上达到当前最优性能
- 适合真实场景下无标注、非对齐数据的匹配任务
构建事件数据与图像之间的像素级对应关系是多传感器系统的基础任务。然而,现有跨模态匹配方法严重依赖匹配标签或严格对齐的硬件,难以适用于无标签且非受控的真实场景,其中既无匹配真值也无先验传感器关系。为此,我们提出一种新型两阶段训练范式:首先利用大规模数据进行无标签蒸馏预训练,通过分布一致性与对比损失提升表示的泛化能力;其次为应对无标签、非约束的下游数据,引入基于极线引导的自蒸馏框架,通过一致性验证筛选稳健匹配,并融合外部极线先验获得几何置信度,使模型可在目标域直接无监督自演化。此外,我们基于TUM-VIE构建了严格的跨模态评估基准,采用物理分离的相机,具有不同的内参与分辨率。大量实验表明,所提方法在MVSEC和TUM-VIE上的位姿估计任务中均达到最先进水平。源代码与基准将公开于https://github.com/ZhonghuaYi/nexus2-official。
原文摘要 · Abstract (English)
Building pixel-level correspondence between event and image data is a fundamental task for multi-sensor systems. However, existing cross-modal matching methods are largely restricted by their reliance on either matching labels or strictly aligned hardware, which limits them to unlabeled and unconstrained real-world scenarios where neither matching ground truth nor prior sensor relationships are available. To address this, we propose a novel two-stage training paradigm. First, we leverage large-scale data to perform label-agnostic distillation pretraining, upgrading optimization objectives with distribution-based and contrastive losses to learn highly generalizable representations. Second, to tackle unlabeled and unconstrained downstream data, we introduce an epipolar-guided self-distillation framework. By utilizing consistency verification to isolate robust matches and incorporating geometric confidence derived from an external epipolar prior, our model can effectively self-evolve directly on target domains without any supervision. Furthermore, we introduce a rigorous cross-modal evaluation benchmark based on TUM-VIE, featuring physically separated cameras with distinct intrinsic parameters and resolutions. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance on both MVSEC and TUM-VIE pose estimation tasks. The source code and benchmark will be made publicly available at https://github.com/ZhonghuaYi/nexus2-official.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。