无需提示词的通用医学图像分割框架，提升自动化与跨模态泛化能力。
Concept-to-Pixel: Prompt-Free Universal Medical Image Segmentation
- 将解剖知识拆分为语义与几何两部分，通过可学习令牌增强模型理解
- 在8个数据集7种模态上实现零样本和跨模态任务的优异表现
- 适合需要自动化、少人工干预的医学图像分析场景
通用医学图像分割旨在使用单一基础模型处理多种成像模态下的多样任务。然而，现有方法常依赖人工视觉提示或检索参考图像，限制了自动化与鲁棒性；且不同模态间的联合训练易受显著域偏移影响。为此，我们提出概念到像素（C2P）——一种新型无提示通用分割框架。C2P将解剖知识显式分解为几何与语义两个组件：利用多模态大语言模型（MLLMs）提炼抽象高阶医学概念为可学习的语义令牌，并引入显式监督的几何令牌以施加通用物理与结构约束。这些解耦令牌与图像特征深度交互，生成输入相关的动态卷积核，实现精确掩码预测。此外，我们提出一种几何感知推理一致性机制，利用模型预测的几何约束评估结果可靠性并抑制异常值。在涵盖七种模态、八个多样化数据集的统一基准上进行的大量实验表明，该联合训练方法显著优于通用或单模型方案。值得注意的是，该统一模型展现出强大泛化能力，在未见病例的零样本任务及相似任务的跨模态迁移中均取得优异效果。代码已公开于：https://github.com/Yundi218/Concept-to-Pixel。
原文摘要 · Abstract (English)
Universal medical image segmentation seeks to use a single foundational model to handle diverse tasks across multiple imaging modalities. However, existing approaches often rely heavily on manual visual prompts or retrieved reference images, which limits their automation and robustness. In addition, naive joint training across modalities often fails to address large domain shifts. To address these limitations, we propose Concept-to-Pixel (C2P), a novel prompt-free universal segmentation framework. C2P explicitly separates anatomical knowledge into two components: Geometric and Semantic representations. It leverages Multimodal Large Language Models (MLLMs) to distill abstract, high-level medical concepts into learnable Semantic Tokens and introduces explicitly supervised Geometric Tokens to enforce universal physical and structural constraints. These disentangled tokens interact deeply with image features to generate input-specific dynamic kernels for precise mask prediction. Furthermore, we introduce a Geometry-Aware Inference Consensus mechanism, which utilizes the model's predicted geometric constraints to assess prediction reliability and suppress outliers. Extensive experiments and analysis on a unified benchmark comprising eight diverse datasets across seven modalities demonstrate the significant superiority of our jointly trained approach, compared to universe- or single-model approaches. Remarkably, our unified model demonstrates strong generalization, achieving impressive results not only on zero-shot tasks involving unseen cases but also in cross-modal transfers across similar tasks. Code is available at: https://github.com/Yundi218/Concept-to-Pixel
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。