arXiv:2607.17341cs.CV2026-07

让医疗眼镜视频分割更准,靠多智能体理解医生视角意图

Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation

论文配图:Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation
图 1 · 摘自论文原文
  • 用双流程多智能体系统,从医生视角理解指令与目标
  • 平均Dice达71.34%,远超基线模型的11.70%
  • 适合临床辅助、智能眼镜等实时医学图像交互场景

交互式第一人称医学图像分割(IEMIS)在智能眼镜辅助医疗影像审阅中至关重要,能从医生的第一人称视角中分割出其关注的医学目标。一旦成功,可提供目标级视觉证据,支持精准分析与临床决策。然而,指令与视频均来自用户第一人称视角,带来两大挑战:(1)语义模糊导致模型难以确认目标;(2)视觉变化使分割结果帧间跳变。本文提出EgoMed-Agent多智能体系统,通过两条工作流实现人类视角理解:(1)目标确认流程基于可靠性得分,将指令与候选目标对齐,可靠时确认目标,不可靠则提示澄清;(2)定位引导传播流程将掩码传播与每帧目标定位耦合,当两者偏离时用定位结果修正掩码,确保跨视频帧稳定分割。大量实验表明,EgoMed-Agent达到71.34%平均Dice,显著高于最佳文本提示基线(11.70%)。代码已公开于项目页。

原文摘要 · Abstract (English)

Interactive egocentric medical image segmentation (IEMIS) plays an important role in smart-glasses-assisted medical image review, segmenting the medical targets a clinician refers to from their egocentric view. Once it succeeds, the object-level visual evidence it provides strengthens the review and underpins fine-grained analysis and clinical decision-making. However, the instruction and the video both come from the user's egocentric perspective, which poses two challenges. (1) Semantic ambiguity leaves the model unable to confirm the user-intended target. (2) Visual variability makes the segmentation jump from frame to frame. In this paper, we propose EgoMed-Agent, a multi-agent system that understands the target from the human perspective through two workflows. (1) The \textit{Target Confirmation Workflow} grounds the instruction against candidate targets with a reliability score, confirming the target when the grounding is reliable and asking the user to clarify when it is not, thereby confirming the segmentation target. (2) The \textit{Localization-Guided Propagation Workflow} couples mask propagation with per-frame target localization, using the localized target to correct the propagated mask whenever the two diverge, so the segmentation stays on the target across the egocentric video. Extensive experiments show that EgoMed-Agent reaches 71.34\% average Dice, far above the best text-prompted baseline (11.70\%). Our code is available at \href{https://github.com/wdyyyyyy/EgoMed-Agent}{our project page}.

医学图像第一人称多智能体分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。