用自回归预测方式统一处理医学图像与文本,实现更简洁高效的病灶分割。
Medical Referring Image Segmentation via Next-Token Mask Prediction
- 将图文分割转为统一序列的下一个词预测任务,无需多模态融合模块。
- 在两个数据集上达到新最好效果,精度提升显著,尤其对细微病灶边界更敏感。
- 适合追求轻量化设计和高精度医学图像理解的研究者或临床辅助系统开发。
医学指称图像分割(MRIS)旨在根据自然语言描述定位医学图像中的目标区域。现有方法通常依赖复杂的多模态融合结构或多阶段解码器。本文提出NTP-MRISeg,将MRIS重构为基于图像、文本与掩码标记序列的自回归下一个标记预测任务。该设计简化模型架构,避免了模态特异性融合与外部分割模型,支持端到端训练。同时利用大规模多模态预训练标记器,增强泛化能力。针对该框架下的暴露偏差、长尾标记分布及细粒度病灶边缘问题,提出三项策略:(1) 下k个标记预测(NkTP)减少累积误差;(2) 标记级对比学习(TCL)提升边界敏感性并缓解长尾分布;(3) 基于记忆的困难标记优化(HET)强化难点标记学习。在QaTa-COV19与MosMedData+数据集上的大量实验表明,NTP-MRISeg取得新最佳性能,为传统MRIS流程提供高效替代方案。
原文摘要 · Abstract (English)
Medical Referring Image Segmentation (MRIS) involves segmenting target regions in medical images based on natural language descriptions. While achieving promising results, recent approaches usually involve complex design of multimodal fusion or multi-stage decoders. In this work, we propose NTP-MRISeg, a novel framework that reformulates MRIS as an autoregressive next-token prediction task over a unified multimodal sequence of tokenized image, text, and mask representations. This formulation streamlines model design by eliminating the need for modality-specific fusion and external segmentation models, supports a unified architecture for end-to-end training. It also enables the use of pretrained tokenizers from emerging large-scale multimodal models, enhancing generalization and adaptability. More importantly, to address challenges under this formulation-such as exposure bias, long-tail token distributions, and fine-grained lesion edges-we propose three novel strategies: (1) a Next-k Token Prediction (NkTP) scheme to reduce cumulative prediction errors, (2) Token-level Contrastive Learning (TCL) to enhance boundary sensitivity and mitigate long-tail distribution effects, and (3) a memory-based Hard Error Token (HET) optimization strategy that emphasizes difficult tokens during training. Extensive experiments on the QaTa-COV19 and MosMedData+ datasets demonstrate that NTP-MRISeg achieves new state-of-the-art performance, offering a streamlined and effective alternative to traditional MRIS pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。