arXiv:2507.23402cs.CVcs.AI2025-07

提出自适应分组对齐框架,提升医学图文跨模态表示学习效果

AGA: An adaptive group alignment framework for structured medical cross-modal representation learning

  • 基于稀疏相似矩阵构建双向分组机制,动态聚合图文语义单元
  • 在多个数据集上实现超越基线的图文检索与分类性能,零样本下仍具优势
  • 适合医学图像分析、跨模态学习等领域的研究者和临床应用开发者

从配对的医学图像与报告中学习视觉表征是表示学习的一个有前景方向。然而,当前医疗领域视觉-语言预训练方法常将临床报告简化为单一实体或碎片化标记,忽视其固有结构。此外,对比学习框架通常依赖大量硬负样本,在小规模医疗数据集上不切实际。为此,我们提出自适应分组对齐(AGA)框架,以捕捉配对医学图像与报告中的结构化语义。AGA引入基于稀疏相似矩阵的双向分组机制:对每对图像-报告,计算文本标记与图像块之间的细粒度相似性;每个标记选择其最匹配的前m个块形成视觉组,每个块选择最相关的标记形成语言组。为实现自适应分组,设计了语言组阈值门控和视觉组阈值门控模块,动态学习分组阈值。组表征通过相似度加权平均计算。为对齐每个标记与其组表征,引入实例感知组对齐损失,该损失在每对图像-文本内部运行,无需外部负样本。最后,应用双向跨模态分组对齐模块,增强视觉与语言组表征间的细粒度对齐。在公开和私有数据集上的大量实验表明,该方法在微调与零样本设置下的图像-文本检索和分类任务中均取得优异性能。

原文摘要 · Abstract (English)

Learning medical visual representations from paired images and reports is a promising direction in representation learning. However, current vision-language pretraining methods in the medical domain often simplify clinical reports into single entities or fragmented tokens, ignoring their inherent structure. In addition, contrastive learning frameworks typically depend on large quantities of hard negative samples, which is impractical for small-scale medical datasets. To tackle these challenges, we propose Adaptive Grouped Alignment (AGA), a new framework that captures structured semantics from paired medical images and reports. AGA introduces a bidirectional grouping mechanism based on a sparse similarity matrix. For each image-report pair, we compute fine-grained similarities between text tokens and image patches. Each token selects its top-matching patches to form a visual group, and each patch selects its most related tokens to form a language group. To enable adaptive grouping, we design two threshold gating modules, called Language Grouped Threshold Gate and Vision Grouped Threshold Gate, which learn grouping thresholds dynamically. Group representations are computed as weighted averages based on similarity scores. To align each token with its group representation, we introduce an Instance Aware Group Alignment loss that operates within each image-text pair, removing the need for external negatives. Finally, a Bidirectional Cross-modal Grouped Alignment module is applied to enhance fine-grained alignment between visual and linguistic group representations. Extensive experiments on public and private datasets show that our method achieves strong performance on image-text retrieval and classification tasks under both fine-tuning and zero-shot settings.

跨模态学习医学影像表示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。