用场景图解析超声图像,帮普通人看懂并指导扫描
Semantic Scene Graph for Ultrasound Image Explanation and Scanning Guidance
- 用Transformer直接生成超声图像场景图,无需先检测物体
- 结合用户提问优化场景图,实现可理解的图像解释
- 能提示遗漏解剖结构,适合非专业人员的临床扫查引导
由于成像和采集参数差异导致的视觉变化,医学超声图像理解仍是长期挑战。尽管大语言模型(LLMs)已用于生成面向临床专家的术语丰富摘要,但对非专业人士(如床旁场景)日益增长的可解释性与基础扫查指导需求尚未得到探索。本研究首次引入超声图像场景图(SG),用于向普通用户解释图像内容并提供扫查指导。采用基于Transformer的一阶段方法直接构建超声SG,避免显式目标检测。通过用户提问进一步利用LLMs精炼抽象的SG表示,生成易于理解的图像解释。此外,预测的SG被用于识别当前视野中缺失的解剖结构,辅助普通用户完成更标准、完整的解剖探查。该方法在5名志愿者的左右颈区(包括颈动脉和甲状腺)超声图像上验证有效,展示了极大提升超声可解释性与可用性的潜力,推动其向大众普及。
原文摘要 · Abstract (English)
Understanding medical ultrasound imaging remains a long-standing challenge due to significant visual variability caused by differences in imaging and acquisition parameters. Recent advancements in large language models (LLMs) have been used to automatically generate terminology-rich summaries orientated to clinicians with sufficient physiological knowledge. Nevertheless, the increasing demand for improved ultrasound interpretability and basic scanning guidance among non-expert users, e.g., in point-of-care settings, has not yet been explored. In this study, we first introduce the scene graph (SG) for ultrasound images to explain image content to ordinary and provide guidance for ultrasound scanning. The ultrasound SG is first computed using a transformer-based one-stage method, eliminating the need for explicit object detection. To generate a graspable image explanation for ordinary, the user query is then used to further refine the abstract SG representation through LLMs. Additionally, the predicted SG is explored for its potential in guiding ultrasound scanning toward missing anatomies within the current imaging view, assisting ordinary users in achieving more standardized and complete anatomical exploration. The effectiveness of this SG-based image explanation and scanning guidance has been validated on images from the left and right neck regions, including the carotid and thyroid, across five volunteers. The results demonstrate the potential of the method to maximally democratize ultrasound by enhancing its interpretability and usability for ordinaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。