用合成描述增强伤口图像,提升糖尿病足溃疡感染识别准确率
Explainable, Multi-modal Wound Infection Classification from Images Augmented with Generated Captions
- 用GPT-4o生成的文本描述增强伤口图像,构建多模态输入
- 在公开数据集上达到85%敏感度、78%特异度和81%准确率
- 生成的描述+可视化结果让护士更易理解诊断依据
糖尿病足溃疡(DFUs)感染可导致组织坏死和截肢,亟需及时准确诊断。现有机器学习方法仅依赖伤口图像,未利用病历等附加信息。本文提出SCARWID框架,通过合成文本描述增强图像以提升感染检测性能。该框架包含:(1) 在GPT-4o生成描述上微调的Wound-BLIP视觉语言模型,用于从图像生成一致的合成描述;(2) 利用交叉注意力融合图像与对应描述的跨模态嵌入。感染状态通过检索带标签支持集中的前k个最相似样本确定。为增加训练数据多样性,使用潜在扩散模型生成额外伤口图像。实验表明,SCARWID在感染分类任务中平均敏感度、特异度和准确率分别达0.85、0.78和0.81,优于当前最优模型。将生成的描述与图像及诊断结果一同展示,提升了模型可解释性与可信度,尤其适用于病历缺失或新手护士辅助诊断场景。
原文摘要 · Abstract (English)
Infections in Diabetic Foot Ulcers (DFUs) can cause severe complications, including tissue death and limb amputation, highlighting the need for accurate, timely diagnosis. Previous machine learning methods have focused on identifying infections by analyzing wound images alone, without utilizing additional metadata such as medical notes. In this study, we aim to improve infection detection by introducing Synthetic Caption Augmented Retrieval for Wound Infection Detection (SCARWID), a novel deep learning framework that leverages synthetic textual descriptions to augment DFU images. SCARWID consists of two components: (1) Wound-BLIP, a Vision-Language Model (VLM) fine-tuned on GPT-4o-generated descriptions to synthesize consistent captions from images; and (2) an Image-Text Fusion module that uses cross-attention to extract cross-modal embeddings from an image and its corresponding Wound-BLIP caption. Infection status is determined by retrieving the top-k similar items from a labeled support set. To enhance the diversity of training data, we utilized a latent diffusion model to generate additional wound images. As a result, SCARWID outperformed state-of-the-art models, achieving average sensitivity, specificity, and accuracy of 0.85, 0.78, and 0.81, respectively, for wound infection classification. Displaying the generated captions alongside the wound images and infection detection results enhances interpretability and trust, enabling nurses to align SCARWID outputs with their medical knowledge. This is particularly valuable when wound notes are unavailable or when assisting novice nurses who may find it difficult to identify visual attributes of wound infection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。