用Transformer提升呼吸音分析准确率,实现更客观的肺部疾病筛查。
Transformer Architectures for Respiratory Sound Analysis and Multimodal Diagnosis
- 采用预训练Audio Spectrogram Transformer模型,结合自注意力机制分析呼吸音。
- 在哮喘检测中达到97%准确率、97% F1分数和0.98 AUC,显著优于传统方法。
- 首次将临床信息融合进诊断流程,适合开发智能辅助诊疗系统。
呼吸音分析是筛查哮喘等肺部疾病的重要手段,但传统听诊主观性强、依赖经验。此前研究基于DenseNet201构建了CNN基准模型,表现良好。本文(i)将Audio Spectrogram Transformer(AST)应用于呼吸音分析,并(ii)评估一个整合声谱图与结构化患者信息的多模态视觉-语言模型(VLM)。AST使用公开预训练权重,在包含每类诊断数百条录音的医疗数据集上微调。VLM采用轻量级Moondream型模型,输入声谱图图像及性别、年龄、采集部位等文本提示,输出JSON格式诊断结果。实验显示,AST在哮喘检测中达到约97%准确率、97% F1分数和0.98 ROC AUC,显著优于内部CNN基线及常见外部基准;VLM达到86%-87%准确率,与CNN相当,且能有效融合临床上下文信息。结果验证了自注意力机制在声学筛查中的有效性,凸显多模态架构在构建综合诊断工具方面的潜力。
原文摘要 · Abstract (English)
Respiratory sound analysis is a crucial tool for screening asthma and other pulmonary pathologies, yet traditional auscultation remains subjective and experience-dependent. Our prior research established a CNN baseline using DenseNet201, which demonstrated high sensitivity in classifying respiratory sounds. In this work, we (i) adapt the Audio Spectrogram Transformer (AST) for respiratory sound analysis and (ii) evaluate a multimodal Vision-Language Model (VLM) that integrates spectrograms with structured patient metadata. AST is initialized from publicly available weights and fine-tuned on a medical dataset containing hundreds of recordings per diagnosis. The VLM experiment uses a compact Moondream-type model that processes spectrogram images alongside a structured text prompt (sex, age, recording site) to output a JSON-formatted diagnosis. Results indicate that AST achieves approximately 97% accuracy with an F1-score around 97% and ROC AUC of 0.98 for asthma detection, significantly outperforming both the internal CNN baseline and typical external benchmarks. The VLM reaches 86-87% accuracy, performing comparably to the CNN baseline while demonstrating the capability to integrate clinical context into the inference process. These results confirm the effectiveness of self-attention for acoustic screening and highlight the potential of multimodal architectures for holistic diagnostic tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。