融合多部位生物信号与动作数据，提升虚拟现实中的情绪识别准确率。
VR Based Emotion Recognition Using Deep Multimodal Fusion With Biosignals Across Multiple Anatomical Domains
- 采用多尺度注意力LSTM与SE模块，融合头、躯干、四肢的多模态信号。
- 23人实验验证，对情感高低维度分类准确率显著提升。
- 揭示不同身体部位信号对情绪识别的贡献差异，适合真实场景应用。
情绪识别通过整合多个解剖区域的多模态生物信号和IMU数据得到显著增强。本文提出一种新型多尺度注意力机制的LSTM架构，结合挤压-激励（SE）模块，利用视觉刺激诱发情绪时来自头部（Meta Quest Pro VR头显）、躯干（Equivital Vest）和外周（Empatica Embrace Plus）的多域信号。共采集23名参与者的数据，并在每次刺激后获取其自评的效价与唤醒度评分。各模态的LSTM层提取特征，多尺度注意力捕捉细粒度时间依赖性，SE模块在分类前重新校准特征重要性。研究评估了不同身体区域信号在虚拟现实体验中携带的情感信息量，识别出关键生物信号。所提架构在用户研究中验证，对效价与唤醒度（高/低）分类表现优异，证实了多域多模态生物信号（如TEMP、EDA）与IMU数据（如加速度计）融合在真实应用场景中的有效性。
原文摘要 · Abstract (English)
Emotion recognition is significantly enhanced by integrating multimodal biosignals and IMU data from multiple domains. In this paper, we introduce a novel multi-scale attention-based LSTM architecture, combined with Squeeze-and-Excitation (SE) blocks, by leveraging multi-domain signals from the head (Meta Quest Pro VR headset), trunk (Equivital Vest), and peripheral (Empatica Embrace Plus) during affect elicitation via visual stimuli. Signals from 23 participants were recorded, alongside self-assessed valence and arousal ratings after each stimulus. LSTM layers extract features from each modality, while multi-scale attention captures fine-grained temporal dependencies, and SE blocks recalibrate feature importance prior to classification. We assess which domain's signals carry the most distinctive emotional information during VR experiences, identifying key biosignals contributing to emotion detection. The proposed architecture, validated in a user study, demonstrates superior performance in classifying valance and arousal level (high / low), showcasing the efficacy of multi-domain and multi-modal fusion with biosignals (e.g., TEMP, EDA) with IMU data (e.g., accelerometer) for emotion recognition in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。