用音频模型检测新冠,发现数据偏差会虚高效果,真实表现很差。
Fine-tuning Pre-trained Audio Models for COVID-19 Detection: A Technical Report
- 用年龄性别分层防数据偏见,让评估更真实
- 跨数据集测试时模型性能暴跌,最高AUC仅0.68
- 小样本加严格控制,难训练出可靠语音新冠检测模型
本技术报告研究预训练音频模型在新冠检测任务上的表现,使用Coswara和COUGHVID两个基准数据集。对Audio-MAE及三种PANN架构(CNN6、CNN10、CNN14)进行微调,并评估了模型在同数据集与跨数据集上的泛化能力。通过按年龄和性别严格分层,防止模型利用人口统计特征与新冠状态之间的虚假关联。同数据集结果中,Audio-MAE在Coswara上表现最佳(AUC 0.82,F1-score 0.76),但所有模型在Coughvid上表现有限(AUC 0.58–0.63)。跨数据集评估显示严重泛化失败(AUC 0.43–0.68),其中Audio-MAE的F1-score降至0.00–0.08。实验表明,尽管人口统计平衡降低了模型表现,却能消除因人口泄漏导致的性能虚高,提供更真实的评估。此外,平衡后数据量仅为1,219–2,160样本,远低于深度学习模型所需规模,难以支撑有效训练。这些发现揭示了构建可泛化语音新冠检测系统的根本挑战,强调在临床评估中需采用严格的分层控制。
原文摘要 · Abstract (English)
This technical report investigates the performance of pre-trained audio models on COVID-19 detection tasks using established benchmark datasets. We fine-tuned Audio-MAE and three PANN architectures (CNN6, CNN10, CNN14) on the Coswara and COUGHVID datasets, evaluating both intra-dataset and cross-dataset generalization. We implemented a strict demographic stratification by age and gender to prevent models from exploiting spurious correlations between demographic characteristics and COVID-19 status. Intra-dataset results showed moderate performance, with Audio-MAE achieving the strongest result on Coswara (0.82 AUC, 0.76 F1-score), while all models demonstrated limited performance on Coughvid (AUC 0.58-0.63). Cross-dataset evaluation revealed severe generalization failure across all models (AUC 0.43-0.68), with Audio-MAE showing strong performance degradation (F1-score 0.00-0.08). Our experiments demonstrate that demographic balancing, while reducing apparent model performance, provides more realistic assessment of COVID-19 detection capabilities by eliminating demographic leakage - a confounding factor that inflate performance metrics. Additionally, the limited dataset sizes after balancing (1,219-2,160 samples) proved insufficient for deep learning models that typically require substantially larger training sets. These findings highlight fundamental challenges in developing generalizable audio-based COVID-19 detection systems and underscore the importance of rigorous demographic controls for clinically robust model evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。