提出双向分步对齐网络,统一解决医学图像配准与融合难题。
BSAFusion: A Bidirectional Stepwise Feature Alignment Network for Unaligned Medical Image Fusion
- 采用双向分步变形场预测,提升跨模态对齐精度。
- 引入无模态差异特征表示,减少模态差异干扰。
- 适合需要高精度多模态医学图像融合的临床研究者。
若能在一个统一框架中通过单阶段方法同时实现未对齐多模态医学图像的配准与融合,不仅可促进两项任务相互增益,还能降低模型复杂度。然而,该设计面临特征融合与对齐需求不兼容的挑战:对齐需对应特征一致,融合则需特征互补。为此,本文提出一种名为双向分步特征对齐与融合(BSFA-F)的未对齐医学图像融合方法。为减轻模态差异对跨模态特征匹配的负面影响,引入无模态差异特征表示(MDF-FR)方法,利用模态特征表示头(MFRH)整合输入图像的全局信息,并将当前图像的MFRH信息注入其他模态图像,有效抑制模态差异影响,同时保留各图像的互补信息。在特征对齐方面,基于两点间向量位移路径无关性,采用双向分步对齐变形场预测策略,解决了单步对齐中跨度大、预测不准的问题。最后,多模态特征融合模块完成对齐特征的融合。多个数据集上的实验结果验证了该方法的有效性。源代码已公开于 https://github.com/slrl123/BSAFusion。
原文摘要 · Abstract (English)
If unaligned multimodal medical images can be simultaneously aligned and fused using a single-stage approach within a unified processing framework, it will not only achieve mutual promotion of dual tasks but also help reduce the complexity of the model. However, the design of this model faces the challenge of incompatible requirements for feature fusion and alignment; specifically, feature alignment requires consistency among corresponding features, whereas feature fusion requires the features to be complementary to each other. To address this challenge, this paper proposes an unaligned medical image fusion method called Bidirectional Stepwise Feature Alignment and Fusion (BSFA-F) strategy. To reduce the negative impact of modality differences on cross-modal feature matching, we incorporate the Modal Discrepancy-Free Feature Representation (MDF-FR) method into BSFA-F. MDF-FR utilizes a Modality Feature Representation Head (MFRH) to integrate the global information of the input image. By injecting the information contained in MFRH of the current image into other modality images, it effectively reduces the impact of modality differences on feature alignment while preserving the complementary information carried by different images. In terms of feature alignment, BSFA-F employs a bidirectional stepwise alignment deformation field prediction strategy based on the path independence of vector displacement between two points. This strategy solves the problem of large spans and inaccurate deformation field prediction in single-step alignment. Finally, Multi-Modal Feature Fusion block achieves the fusion of aligned features. The experimental results across multiple datasets demonstrate the effectiveness of our method. The source code is available at https://github.com/slrl123/BSAFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。