用文字指令控制医学图像编辑,提升息肉和皮肤病变分割效果
InstEditSeg: Instruction-Driven Image Editing for Polyp and Skin Lesion Segmentation

- 将分割任务转为图文驱动的图像编辑,生成符合自然图像分布的彩色叠加图
- 在肠镜和皮肤镜数据集上达到与顶尖判别模型相当的精度,跨域泛化更强
- 支持多病灶完整分割、指令控制和灵活采样,适合临床辅助诊断场景
准确分割息肉和皮肤病变对临床诊断至关重要,但现有方法在低对比度、边界模糊及跨域分布差异方面表现不佳。主流判别网络与扩散模型通常输出独立二值掩码,未能利用大规模预训练生成模型的视觉先验。本文提出InstEditSeg,一个统一的生成式框架,将医学分割重构为指令驱动的图像编辑问题。模型不直接输出掩码,而是基于文本指令在原始图像上生成彩色叠加图,使编辑结果契合潜在扩散模型学习到的自然图像分布,缓解自然图像与医学影像间的域差距。为恢复精细解剖结构,引入DINOv3作为辅助视觉编码器,并设计多尺度特征金字塔与零初始化卷积融合模块,将层次化判别先验注入扩散U-Net而不扰动预训练权重。采用仅需两次前向传播的双分支无分类器引导策略,降低推理开销。在息肉和皮肤病变基准测试中,该框架性能媲美强判别基线,且展现生成范式的显著优势:对未见数据具备更优跨域泛化能力、更完整的多病灶分割、指令可控的任务操作以及采样灵活性。同时分析了该范式局限,包括颜色敏感性及不支持属性条件选择。代码已公开:https://github.com/wincharm001/InstEditSeg。
原文摘要 · Abstract (English)
Accurate segmentation of polyps and skin lesions is pivotal for clinical diagnosis, yet existing methods struggle with low contrast, ambiguous boundaries, and cross-domain distribution discrepancies. Discriminative networks and most diffusion-based segmentation approaches predict standalone binary masks, leaving the visual priors of large-scale pretrained generative models largely unexploited. We propose InstEditSeg, a unified generative framework that reformulates medical segmentation as an instruction-driven image editing problem. Instead of emitting a mask, the model renders a color-coded overlay on the original image, conditioned on a textual instruction, so that the edited output aligns with the natural image distribution learned by latent diffusion models and mitigates the domain gap between natural and medical imagery. To recover fine anatomical structures, we introduce DINOv3 as an auxiliary visual encoder and a DINO Feature Guidance Block that builds a multi-scale feature pyramid. The pyramid is fused into the diffusion U-Net by channel concatenation and zero-initialized convolution so that hierarchical discriminative priors can be injected without perturbing the pretrained weights. A dual-branch classifier-free guidance strategy requiring only two forward passes per denoising step reduces inference cost. On polyp and skin lesion benchmarks the framework achieves accuracy competitive with strong discriminative baselines, and it further demonstrates concrete advantages of the generative formulation: notably better cross-domain generalization on unseen data, more complete multi-lesion segmentation, instruction-conditioned task control, and sampling flexibility. We also analyze the strengths and limitations of the paradigm, including its color sensitivity and unsupported attribute-conditioned selection. Code is available at: https://github.com/wincharm001/InstEditSeg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。