用AI融合表情、语音和动作,快速判断中风,准确率达95.8%
Digital FAST: An AI-Driven Multimodal Framework for Rapid and Early Stroke Screening
- 融合面部、语音、肢体三模态数据,用Transformer等模型捕捉时序特征
- 在222段视频上达95.83%准确率,所有中风病例均被检出
- 适合急救场景快速筛查,推动AI辅助中风早期诊断
早期识别中风症状对及时干预和改善患者预后至关重要,尤其在院前环境中。本研究提出一种基于F.A.S.T.评估数据的快速、无创多模态深度学习框架,用于自动二分类中风筛查。该方法整合面部表情、语音信号和上肢运动的互补信息以增强诊断鲁棒性。面部动态通过基于关键点的特征表示,并使用Transformer架构捕捉时序依赖;语音信号转换为梅尔频谱图,由音频频谱变换器处理;上肢姿态序列则通过MLP-Mixer网络建模时空运动模式。各模态特异性表征通过注意力融合机制结合,有效学习跨模态交互。在自收集的222个视频(来自37名受试者)上的实验表明,所提多模态模型持续优于单模态基线,达到95.83%准确率和96.00% F1分数,兼具高敏感性和特异性,测试集中所有中风病例均被成功检测。结果凸显多模态学习与迁移学习在早期中风筛查中的潜力,同时强调需更大、更具临床代表性的数据集支持真实世界部署。
原文摘要 · Abstract (English)
Early identification of stroke symptoms is essential for enabling timely intervention and improving patient outcomes, particularly in prehospital settings. This study presents a fast, non-invasive multimodal deep learning framework for automatic binary stroke screening based on data collected during the F.A.S.T. assessment. The proposed approach integrates complementary information from facial expressions, speech signals, and upper-body movements to enhance diagnostic robustness. Facial dynamics are represented using landmark based features and modeled with a Transformer architecture to capture temporal dependencies. Speech signals are converted into mel spectrograms and processed using an Audio Spectrogram Transformer, while upper-body pose sequences are analyzed with an MLP-Mixer network to model spatiotemporal motion patterns. The extracted modality specific representations are combined through an attention-based fusion mechanism to effectively learn cross modal interactions. Experiments conducted on a self-collected dataset of 222 videos from 37 subjects demonstrate that the proposed multimodal model consistently outperforms unimodal baselines, achieving 95.83% accuracy and a 96.00% F1-score. The model attains a strong balance between sensitivity and specificity and successfully detects all stroke cases in the test set. These results highlight the potential of multimodal learning and transfer learning for early stroke screening, while emphasizing the need for larger, clinically representative datasets to support reliable real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。