MedPixel统一医学图像与语言理解,实现像素级精准定位。
MedPixel: A Unified Pixel-Language Model for Medical Reasoning and Segmentation

- 构建语言-掩码共享接口,实现视觉与语言的像素级对齐。
- 在44万样本上训练,零样本迁移表现优异,抗不完美提示鲁棒。
- 适合医学影像分析、智能诊断等需要细粒度理解的场景。
可靠的医学图像理解需连接临床语言与视觉推理,并实现像素级定位。然而,现有医学视觉-语言模型常缺乏精确空间定位,而医学分割模型多依赖明确类别或精准空间提示。这一差距源于监督不匹配:分割数据集提供精细掩码但缺乏语言监督,医疗视觉-语言数据则极少包含密集空间标注。为此,我们提出MedPixel,一种基于共享语言-掩码接口的统一医学像素-语言模型。为实现可扩展监督,我们构建了约44万条像素-语言任务样本的MedPLG-440K,通过临床驱动的合成过程生成,无需外部大语言模型标注。MedPixel采用联合多任务监督微调,再经像素级偏好优化,利用真实掩码作为离线验证器,从掩码质量推导响应偏好。该模型支持显式定位、隐式推理、空间交互、接地解释和医学VQA等多种任务,在像素预测与回答生成上均表现强劲,并具备良好的零样本迁移能力及对不完整空间提示的鲁棒性。代码与模型权重将开源于https://github.com/yhy-whu/Medpixel。
原文摘要 · Abstract (English)
Reliable medical image understanding requires models to connect clinical language and visual reasoning with pixel-level grounding. Yet medical vision-language models often lack precise localization, whereas medical segmenters typically rely on explicit target categories or precise spatial prompts. This divide is reinforced by a supervision mismatch: segmentation datasets provide precise masks but little language supervision, whereas medical vision-language data rarely pair language with dense spatial annotations. To address this gap, we present MedPixel, a unified medical pixel-language model built around a shared language--mask interface. To provide scalable supervision, we introduce MedPLG-440K, comprising approximately 440K pixel-language task samples constructed through a clinically motivated synthesis process without external LLM annotation. MedPixel is trained with joint multi-task supervised fine-tuning followed by Pixel-Level Preference Optimization, which uses ground-truth masks as offline verifiers to derive response preferences from mask quality. MedPixel supports a broad spectrum of tasks spanning explicit grounding, implicit reasoning, spatial interaction, grounded explanation, and medical VQA. Across this task spectrum, MedPixel achieves strong performance in both pixel-level prediction and response generation, together with effective zero-shot transfer to external grounding benchmarks and robustness to imperfect spatial prompts. Code and model checkpoints will be released at https://github.com/yhy-whu/Medpixel.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。