用多模态Transformer识别社交媒体中的抑郁迹象,效果优于现有方法。
MMFformer: Multimodal Fusion Transformer Network for Depression Detection
- 设计双分支Transformer提取视频空间特征与音频时序动态
- 通过中间与晚期融合策略捕捉跨模态相关性,提升检测精度
- 在两个大规模数据集上显著超越主流方法,适合心理健康研究者
抑郁症是一种严重影响个体身心健康和生活质量的严重精神疾病,早期检测对及时干预至关重要。由于诊断主要依赖临床访谈中的主观评估,因此借助社交媒体内容实现早期筛查成为重要研究方向。用户生成内容具有广泛性和多样性,给有效提取时空信息及多模态数据融合带来挑战。本文提出MMFformer,一种用于抑郁检测的多模态融合Transformer网络,旨在从多模态社交媒体信息中挖掘抑郁相关的时空高级模式。带有残差连接的Transformer网络捕获视频的空间特征,而变压器编码器则用于建模音频的重要时序动态。此外,融合架构采用中间与晚期融合策略,以发现不同模态间的最相关关联。最终,在两个大规模抑郁检测数据集上评估该网络,结果表明其性能显著优于现有最先进方法:在D-Vlog数据集上F1分数提升13.92%,在LMVD数据集上提升7.74%。代码已公开于https://github.com/rezwanh001/Large-Scale-Multimodal-Depression-Detection。
原文摘要 · Abstract (English)
Depression is a serious mental health illness that significantly affects an individual's well-being and quality of life, making early detection crucial for adequate care and treatment. Detecting depression is often difficult, as it is based primarily on subjective evaluations during clinical interviews. Hence, the early diagnosis of depression, thanks to the content of social networks, has become a prominent research area. The extensive and diverse nature of user-generated information poses a significant challenge, limiting the accurate extraction of relevant temporal information and the effective fusion of data across multiple modalities. This paper introduces MMFformer, a multimodal depression detection network designed to retrieve depressive spatio-temporal high-level patterns from multimodal social media information. The transformer network with residual connections captures spatial features from videos, and a transformer encoder is exploited to design important temporal dynamics in audio. Moreover, the fusion architecture fused the extracted features through late and intermediate fusion strategies to find out the most relevant intermodal correlations among them. Finally, the proposed network is assessed on two large-scale depression detection datasets, and the results clearly reveal that it surpasses existing state-of-the-art approaches, improving the F1-Score by 13.92% for D-Vlog dataset and 7.74% for LMVD dataset. The code is made available publicly at https://github.com/rezwanh001/Large-Scale-Multimodal-Depression-Detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。