arXiv:2507.12236cs.CV2025-07被引 2

用生成式模型提升医学影像中的短语定位精度,效果远超传统方法。

Generate to Ground: Multimodal Text Conditioning Boosts Phrase Grounding in Medical Vision-Language Models

  • 基于扩散模型的生成式方法,利用跨注意力图实现短语定位。
  • 零样本下mIoU翻倍,显著优于现有判别式方法。
  • 适合医疗视觉语言模型研究者与临床辅助诊断系统开发者。

短语定位(phrase grounding)即把自然语言短语映射到图像特定区域,在临床报告指导下对医学影像中的疾病位置定位具有重要意义。现有最优方法依赖判别式、自监督对比模型,本文证明生成式文本到图像扩散模型通过利用跨注意力图,可实现更优的零样本短语定位性能。与以往认知相反,我们发现使用冻结的领域特定语言模型(如CXR-BERT)微调扩散模型,显著优于通用模型。该设置使mIoU得分翻倍。为进一步提升性能,提出双模态偏差融合(BBM)后处理技术,对齐文本与图像偏差以识别高置信度区域,优化跨注意力图,进一步提高定位精度。结果表明生成式方法在医学影像短语定位中更具优势,为临床应用提供更鲁棒、可解释的解决方案。代码与模型权重已公开于https://github.com/Felix-012/generate_to_ground。

原文摘要 · Abstract (English)

Phrase grounding, i.e., mapping natural language phrases to specific image regions, holds significant potential for disease localization in medical imaging through clinical reports. While current state-of-the-art methods rely on discriminative, self-supervised contrastive models, we demonstrate that generative text-to-image diffusion models, leveraging cross-attention maps, can achieve superior zero-shot phrase grounding performance. Contrary to prior assumptions, we show that fine-tuning diffusion models with a frozen, domain-specific language model, such as CXR-BERT, substantially outperforms domain-agnostic counterparts. This setup achieves remarkable improvements, with mIoU scores doubling those of current discriminative methods. These findings highlight the underexplored potential of generative models for phrase grounding tasks. To further enhance performance, we introduce Bimodal Bias Merging (BBM), a novel post-processing technique that aligns text and image biases to identify regions of high certainty. BBM refines cross-attention maps, achieving even greater localization accuracy. Our results establish generative approaches as a more effective paradigm for phrase grounding in the medical imaging domain, paving the way for more robust and interpretable applications in clinical practice. The source code and model weights are available at https://github.com/Felix-012/generate_to_ground.

医学影像短语定位生成模型扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。