让大模型像人一样一步步看图切病灶,精准定位医学图像中的目标
IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation
- 用多步推理+点击操作替代传统分割,无需修改模型架构
- 在多个医学数据集上分割准确率超越现有最佳方法
- 适合需要精细定位的临床辅助诊断场景
近期医学多模态大模型研究从图像级理解转向像素级细粒度认知。尽管分割是像素级理解的基础,现有方法存在两大挑战:一是引入隐式分割标记并需同步微调模型与外部解码器,易导致灾难性遗忘且泛化能力差;二是多数方法依赖单次推理,缺乏迭代优化能力,性能受限。为此,我们提出新型代理型多模态大模型 IBISAgent,将分割重构为以视觉为中心的多步决策过程。IBISAgent使模型能生成交替的推理与文本点击动作,调用分割工具,并在不修改架构的情况下生成高质量掩码。通过在掩码图像特征上进行多步迭代视觉推理,自然支持掩码精炼,推动像素级视觉推理能力发展。我们设计了两阶段训练框架:冷启动监督微调与定制化的细粒度奖励强化学习,显著提升模型在复杂医学指代与推理分割任务中的鲁棒性。大量实验表明,IBISAgent持续优于闭源与开源最先进方法。
原文摘要 · Abstract (English)
Recent research on medical MLLMs has gradually shifted its focus from image-level understanding to fine-grained, pixel-level comprehension. Although segmentation serves as the foundation for pixel-level understanding, existing approaches face two major challenges. First, they introduce implicit segmentation tokens and require simultaneous fine-tuning of both the MLLM and external pixel decoders, which increases the risk of catastrophic forgetting and limits generalization to out-of-domain scenarios. Second, most methods rely on single-pass reasoning and lack the capability to iteratively refine segmentation results, leading to suboptimal performance. To overcome these limitations, we propose a novel agentic MLLM, named IBISAgent, that reformulates segmentation as a vision-centric, multi-step decision-making process. IBISAgent enables MLLMs to generate interleaved reasoning and text-based click actions, invoke segmentation tools, and produce high-quality masks without architectural modifications. By iteratively performing multi-step visual reasoning on masked image features, IBISAgent naturally supports mask refinement and promotes the development of pixel-level visual reasoning capabilities. We further design a two-stage training framework consisting of cold-start supervised fine-tuning and agentic reinforcement learning with tailored, fine-grained rewards, enhancing the model's robustness in complex medical referring and reasoning segmentation tasks. Extensive experiments demonstrate that IBISAgent consistently outperforms both closed-source and open-source SOTA methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。