arXiv:2501.07556cs.CV2025-01被引 64

用大规模合成数据训练模型,实现跨模态图像匹配的通用化。

MatchAnything: Universal Cross-Modality Image Matching with Large-Scale Pre-Training

  • 通过合成跨模态数据预训练,学习跨域结构匹配能力。
  • 在8个未见过的跨模态任务上表现显著优于现有方法。
  • 适合多模态医学影像、遥感等需跨域配准的研究者使用。

图像匹配旨在识别图像间对应的像素位置,在图像配准、融合与分析中至关重要。近年来,基于深度学习的方法在快速准确地寻找大量对应点方面已超越人类。然而,当图像来自不同成像模态且外观差异显著时,由于标注的跨模态训练数据稀缺,算法性能通常下降,限制了多模态信息互补应用的发展。为此,我们提出一种大规模预训练框架,利用多样化的合成跨模态训练信号,训练模型识别并匹配图像间的底层结构。该能力可迁移至真实世界中未见过的跨模态图像匹配任务。关键发现是:使用该框架训练的模型在超过八个未见的跨模态配准任务上仅用同一组权重即表现出卓越泛化能力,显著优于现有通用或特定任务设计的方法。这一进展显著提升了图像匹配技术在多个科学领域的适用性,并为多模态人机智能分析开辟新路径。

原文摘要 · Abstract (English)

Image matching, which aims to identify corresponding pixel locations between images, is crucial in a wide range of scientific disciplines, aiding in image registration, fusion, and analysis. In recent years, deep learning-based image matching algorithms have dramatically outperformed humans in rapidly and accurately finding large amounts of correspondences. However, when dealing with images captured under different imaging modalities that result in significant appearance changes, the performance of these algorithms often deteriorates due to the scarcity of annotated cross-modal training data. This limitation hinders applications in various fields that rely on multiple image modalities to obtain complementary information. To address this challenge, we propose a large-scale pre-training framework that utilizes synthetic cross-modal training signals, incorporating diverse data from various sources, to train models to recognize and match fundamental structures across images. This capability is transferable to real-world, unseen cross-modality image matching tasks. Our key finding is that the matching model trained with our framework achieves remarkable generalizability across more than eight unseen cross-modality registration tasks using the same network weight, substantially outperforming existing methods, whether designed for generalization or tailored for specific tasks. This advancement significantly enhances the applicability of image matching technologies across various scientific disciplines and paves the way for new applications in multi-modality human and artificial intelligence analysis and beyond.

图像匹配跨模态预训练医学影像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。