arXiv:2409.02243cs.CV2024-09被引 1

用音视频融合分析情绪特征,提升抑郁症与多动症的自动诊断准确率。

A Novel Audio-Visual Information Fusion System for Mental Disorders Detection

  • 结合语音和面部表情,用时空注意力网络实现多模态融合
  • 在真实ADHD数据集上达到80%以上准确率,优于现有方法
  • 首次统一系统检测多种精神障碍,适合医疗辅助诊断场景

精神障碍是全球医疗体系面临的重要挑战。及时诊断与干预对治疗至关重要,但部分精神障碍的早期躯体症状不明显,常被忽略或误诊。传统诊断方法耗时且成本高。基于fMRI和EEG的深度学习方法虽提升了效率,但设备与专业人员成本高昂,且多数系统仅针对特定疾病,缺乏通用性。近年来生理研究发现,抑郁症、多动症等存在言语与面部表现异常。本文聚焦精神障碍的情绪表达特征,提出一种基于音视频信息输入的多模态诊断系统。该系统采用时空注意力网络,创新性地使用计算量较低的预训练音频识别网络微调视频识别模块,提升性能。首次实现统一系统对多种精神障碍(注意缺陷多动障碍与抑郁症)的检测。在真实多模态ADHD数据集上准确率超过80%,在抑郁数据集AVEC 2014上达到当前最优水平。

原文摘要 · Abstract (English)

Mental disorders are among the foremost contributors to the global healthcare challenge. Research indicates that timely diagnosis and intervention are vital in treating various mental disorders. However, the early somatization symptoms of certain mental disorders may not be immediately evident, often resulting in their oversight and misdiagnosis. Additionally, the traditional diagnosis methods incur high time and cost. Deep learning methods based on fMRI and EEG have improved the efficiency of the mental disorder detection process. However, the cost of the equipment and trained staff are generally huge. Moreover, most systems are only trained for a specific mental disorder and are not general-purpose. Recently, physiological studies have shown that there are some speech and facial-related symptoms in a few mental disorders (e.g., depression and ADHD). In this paper, we focus on the emotional expression features of mental disorders and introduce a multimodal mental disorder diagnosis system based on audio-visual information input. Our proposed system is based on spatial-temporal attention networks and innovative uses a less computationally intensive pre-train audio recognition network to fine-tune the video recognition module for better results. We also apply the unified system for multiple mental disorders (ADHD and depression) for the first time. The proposed system achieves over 80\% accuracy on the real multimodal ADHD dataset and achieves state-of-the-art results on the depression dataset AVEC 2014.

精神障碍检测多模态融合音视频分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。