arXiv:2606.06760cs.CV2026-06中稿 · ICML

让医学视觉语言模型精准定位病灶区域,实现图像与语义的统一理解。

MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models

论文配图:MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
图 1 · 摘自论文原文
  • 引入区域感知模块,将空间信息编码到语言模型表征中。
  • 用解码器重建分割图,实现像素级定位与语义解释的端到端对齐。
  • 仅用7.2万组数据训练,跨模态任务表现领先,适合临床辅助诊断场景。

医学大规模视觉语言模型(Med-LVLMs)在视觉-语言理解与医学图像分割方面取得显著进展,但现有模型仍难以统一这两项能力,而这对于实现连接视觉发现与语义解释的临床推理至关重要。我们提出 MedSIGHT,一个统一框架,使 Med-LVLMs 具备结构化、像素级的理解能力,实现基于视觉的精准理解。MedSIGHT 引入一种新型区域感知模块(Region Perceiver),生成以区域为中心的令牌,将空间信息直接编码至语言模型的表示空间。同时,我们构建了一个医学区域码本(medical region codebook)并融入大语言模型(LLM)词汇表,使模型可生成离散区域代码,作为解剖与病理区域的符号化表达。这些代码通过区域感知模块解码,重建出分割掩码,实现端到端的空间定位。最后,采用渐进式训练策略,逐步稳定对齐区域感知模块、码本与大语言模型。仅在 7.2 万组多模态指令对上训练,MedSIGHT 在多种成像模态下,于医学理解与分割任务中均达到当前最优性能。

原文摘要 · Abstract (English)

Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation. We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.

医学视觉视觉语言模型像素级定位端到端

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。