用大模型分析糖尿病足溃疡图像,自动生成临床描述。
UlcerGPT: A Multimodal Approach Leveraging Large Language and Vision Models for Diabetic Foot Ulcer Image Transcription
- 融合视觉与语言大模型,自动识别并描述溃疡区域。
- 在公开数据集上经专家评估,转录准确率显著提升。
- 适合远程医疗中的医生辅助诊断,尤其资源有限地区。
糖尿病足溃疡(DFUs)是导致住院和下肢截肢的主要原因,给患者和医疗系统带来巨大负担。早期检测和准确分类对预防严重并发症至关重要,但许多患者因难以获得专科服务而延误治疗。远程医疗为改善医疗可及性提供了新途径。人工智能与模式识别技术的引入进一步提升了基于图像的DFU自动检测、分类和监测能力。尽管已有诸多AI方法应用于DFU图像分析,但利用大语言模型进行图像转录的研究尚属空白。为此,我们提出UlcerGPT,一种结合大语言模型与视觉模型的多模态框架,用于糖尿病足溃疡图像的自动转录。该方法通过大型语言与视觉助手(Large Language and Vision Assistant)及Chat Generative Pre-trained Transformer等模型,联合实现病灶区域的检测、分类与定位。在公开数据集上的实验结果表明,经专家临床评估,UlcerGPT在转录准确性与效率方面表现优异,为远程医疗中及时诊疗提供潜在支持。
原文摘要 · Abstract (English)
Diabetic foot ulcers (DFUs) are a leading cause of hospitalizations and lower limb amputations, placing a substantial burden on patients and healthcare systems. Early detection and accurate classification of DFUs are critical for preventing serious complications, yet many patients experience delays in receiving care due to limited access to specialized services. Telehealth has emerged as a promising solution, improving access to care and reducing the need for in-person visits. The integration of artificial intelligence and pattern recognition into telemedicine has further enhanced DFU management by enabling automatic detection, classification, and monitoring from images. Despite advancements in artificial intelligence-driven approaches for DFU image analysis, the application of large language models for DFU image transcription has not yet been explored. To address this gap, we introduce UlcerGPT, a novel multimodal approach leveraging large language and vision models for DFU image transcription. This framework combines advanced vision and language models, such as Large Language and Vision Assistant and Chat Generative Pre-trained Transformer, to transcribe DFU images by jointly detecting, classifying, and localizing regions of interest. Through detailed experiments on a public dataset, evaluated by expert clinicians, UlcerGPT demonstrates promising results in the accuracy and efficiency of DFU transcription, offering potential support for clinicians in delivering timely care via telemedicine.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。