arXiv:2412.19412cs.CV2024-12中稿 · CVPR被引 81

用生成数据提升跨模态图像匹配通用性,效果超越专用模型。

MINIMA: Modality Invariant Image Matching

  • 通过生成模型扩增多模态数据,继承原始RGB数据的标签与多样性
  • 构建包含19种跨模态组合的MD-syn数据集,实现零样本匹配能力
  • 无需复杂模块,仅靠数据规模提升,适合通用图像匹配研究者

跨视图和跨模态的图像匹配在多模态感知中至关重要。现实场景中,不同成像系统带来的模态差异极大增加了匹配难度。现有方法针对特定模态提取不变特征,并在有限数据上训练,泛化能力差。本文提出MINIMA,一个统一的多跨模态图像匹配框架。不依赖复杂模块,而是从数据规模入手提升通用性能。为此,设计了一种简单有效的数据引擎,通过生成模型将廉价丰富的仅RGB匹配数据扩展为包含多种模态、丰富场景且带精确匹配标签的大规模数据。该过程保留了原始RGB数据的标签和多样性。基于此,构建了新数据集MD-syn,填补通用多模态图像匹配的数据空白。借助MD-syn,可直接在任意模态对上训练先进匹配流水线以获得跨模态能力。在19种跨模态设置下的域内与零样本匹配任务中,大量实验表明MINIMA显著优于基线,甚至超越专用方法。代码与数据已开源。

原文摘要 · Abstract (English)

Image matching for both cross-view and cross-modality plays a critical role in multimodal perception. In practice, the modality gap caused by different imaging systems/styles poses great challenges to the matching task. Existing works try to extract invariant features for specific modalities and train on limited datasets, showing poor generalization. In this paper, we present MINIMA, a unified image matching framework for multiple cross-modal cases. Without pursuing fancy modules, our MINIMA aims to enhance universal performance from the perspective of data scaling up. For such purpose, we propose a simple yet effective data engine that can freely produce a large dataset containing multiple modalities, rich scenarios, and accurate matching labels. Specifically, we scale up the modalities from cheap but rich RGB-only matching data, by means of generative models. Under this setting, the matching labels and rich diversity of the RGB dataset are well inherited by the generated multimodal data. Benefiting from this, we construct MD-syn, a new comprehensive dataset that fills the data gap for general multimodal image matching. With MD-syn, we can directly train any advanced matching pipeline on randomly selected modality pairs to obtain cross-modal ability. Extensive experiments on in-domain and zero-shot matching tasks, including $19$ cross-modal cases, demonstrate that our MINIMA can significantly outperform the baselines and even surpass modality-specific methods. The dataset and code are available at https://github.com/LSXI7/MINIMA.

图像匹配跨模态数据生成通用性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。