arXiv:2606.03180cs.CVcs.CL2026-06

让医学影像模型精准定位病灶区域,实现无需标注的零样本分割。

GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations

论文配图:GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
图 1 · 摘自论文原文
  • 用稀疏门控机制只激活与文本查询相关的图像片段,避免全局平铺注意力。
  • 在2D X光和3D CT上均实现零样本分割,3D CT零样本分割为首次突破。
  • 适合需要精准定位病灶的临床辅助诊断场景,尤其擅长自由文本查询。

放射科视觉语言模型(VLM)通过自然产生的影像-报告配对实现了可扩展的范式,但存在尺度不匹配问题:每个病灶仅占图像小区域,而监督信号却仅在全局图像-报告层面提供。现有方法将权重均匀分布于所有图像块,而非聚焦于与查询相关的稀疏子集。为此,本文提出GLINT(Gated Language-Image alignmeNT),显式建模这种稀疏对应关系。在对齐侧,引入稀疏门控对齐架构,通过独立门嵌入空间中的Sigmoid门仅激活与文本查询相关的图像块,强制显式稀疏性;在表示侧,加入密集特征正则化,将可训练编码器的中间特征锚定至冻结的自监督学习(SSL)教师模型,保留门控机制所依赖的细粒度图像块特征。该方法适用于2D胸片(CXR)与3D胸部CT,分别基于DINOv3和V-JEPA 2.1构建。GLINT支持自由文本查询下的零样本分类、定位与分割,据我们所知是首个在无掩码监督下实现3D CT体积零样本分割的方法。显著提升出现在零样本定位与分割任务中,与设计初衷一致。下游评估显示,其在分类、报告生成与分割任务上均优于原始SSL编码器及现有医疗VLM。

原文摘要 · Abstract (English)

Vision-language models (VLMs) for radiology have emerged as a scalable paradigm by leveraging image-report pairs naturally produced in clinical workflows. However, this pairing reveals a mismatch in scale: each finding occupies only a small region of the image, yet supervision is provided only at the global image-report level. This poses a central challenge: prior approaches spread weight densely across all patches rather than concentrating on the sparse subset relevant to a given query. To address this, we present GLINT (Gated Language-Image alignmeNT), a framework that explicitly models this sparse correspondence. On the alignment side, we introduce Sparsely Gated Alignment, a novel architecture in which a sigmoid gate over a separate gate embedding space activates only the patches relevant to each textual query, enforcing explicit sparsity. On the representation side, we add Dense Feature Regularization, which anchors the trainable encoder's intermediate features to a frozen self-supervised learning (SSL) teacher, preserving the fine-grained patch features that the gate relies on. The same recipe applies to both 2D chest X-ray (CXR) and 3D chest computed tomography (CT), built with DINOv3 and V-JEPA 2.1, respectively. GLINT enables zero-shot classification, grounding, and segmentation from free-text queries, and to our knowledge is the first to demonstrate zero-shot segmentation on 3D CT volumes without mask supervision. Notably, the most pronounced gains arise on zero-shot grounding and segmentation, where sparse, query-specific localization is required, consistent with our design intent. In downstream evaluation, GLINT outperforms both SSL encoders and medical VLMs on classification, report generation, and segmentation.

医学影像零样本稀疏对齐3D分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。