提出早融合框架，让图文增广不破坏空间对齐。
A Text-Image Fusion Method with Data Augmentation Capabilities for Referring Medical Image Segmentation
- 图文特征先融合再增广，保持空间一致性。
- 生成伪图像准确定位病灶区域，提升分割性能。
- 适配多种医学分割任务，代码开源可复现。
深度学习严重依赖数据增广以缓解医疗影像数据不足问题。近年来多模态学习通过文本与图像结合实现指代性图像分割（referring medical image segmentation）。然而，常规增广方法如旋转、翻转会破坏图像与文本间的空间对齐，导致性能下降。为此，本文提出一种早期融合框架，在增广前融合文本与视觉特征，保持空间一致性。同时设计轻量级生成器，将文本嵌入投影至视觉空间，弥合语义鸿沟。可视化生成的伪图像显示病灶定位准确。方法在三个医学影像任务和四种分割框架上验证，达到当前最优效果。代码已公开于GitHub：https://github.com/11yxk/MedSeg_EarlyFusion。
原文摘要 · Abstract (English)
Deep learning relies heavily on data augmentation to mitigate limited data, especially in medical imaging. Recent multimodal learning integrates text and images for segmentation, known as referring or text-guided image segmentation. However, common augmentations like rotation and flipping disrupt spatial alignment between image and text, weakening performance. To address this, we propose an early fusion framework that combines text and visual features before augmentation, preserving spatial consistency. We also design a lightweight generator that projects text embeddings into visual space, bridging semantic gaps. Visualization of generated pseudo-images shows accurate region localization. Our method is evaluated on three medical imaging tasks and four segmentation frameworks, achieving state-of-the-art results. Code is publicly available on GitHub: https://github.com/11yxk/MedSeg_EarlyFusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。