用多模态数据与可解释分析,提升VR中身体失衡状态的识别准确率。
Toward Postural State Classification in Immersive VR with Multimodal Data and Explainability Analysis

- 融合运动、肌电与皮电信号,用时序深度模型识别身体失衡状态。
- 最佳模型准确率达96.76%,仅用三分之二重要特征仍保持0.957准确率。
- 通过SHAP分析揭示动作模式是判断失衡的关键,适合安全型VR系统设计。
确保沉浸式虚拟现实(VR)体验的安全需要能够预测并响应用户失衡的系统。尽管已有研究关注跌倒预测和晕动症,但多数方法为回归型,姿势状态分类仍较少被探索。本研究对比了机器学习(ML)与深度学习(DL)模型在视觉干扰下对VR中姿势状态的分类能力。使用包含运动学、肌电(EMG)和皮电活动(EDA)信号的多模态数据集,将任务设定为二分类:区分平衡与失衡状态,并采用参与者级下采样缓解类别不平衡问题。所有模型均通过留一参与者交叉验证(LOPO)评估泛化性能。其中,受Mamba启发的卷积神经网络(MI-CNN)达到最高准确率96.76%。通过SHapley加性解释(SHAP)分析提升可解释性,结果显示运动学特征占主导地位,表明身体运动模式对检测VR中失衡具有显著信息量。进一步实验显示,仅使用前三分之二高重要性特征,模型仍维持0.957的准确率与F1分数,较全特征模型下降约1%。结果表明,多模态传感、时序深度学习与可解释人工智能可有效支持VR中平衡相关不稳定性的可靠分类。准确识别失衡状态有助于提升跌倒风险意识,推动更安全、自适应的VR系统发展。代码已公开:https://github.com/NipaAnjum/MI-CNN。
原文摘要 · Abstract (English)
Ensuring a safe virtual reality (VR) experience requires systems that can predict and respond when users lose their balance. Although prior work has examined fall prediction and motion sickness, many approaches are regression-based and postural state classification remains less explored. This study compares machine learning (ML) and deep learning (DL) models for classifying postural states in VR under visual perturbations. We used a multimodal dataset containing kinematic, electromyographic (EMG), and electrodermal activity (EDA) signals. The data were prepared for a binary task to distinguish balanced from imbalanced postural states, and participant-wise downsampling addressed class imbalance. All models were evaluated with Leave-One-Participant-Out (LOPO) cross-validation to test generalization to unseen participants. Among the models, the Mamba-inspired CNN (MI-CNN) achieved the highest accuracy of 96.76%. SHapley Additive exPlanations (SHAP) analysis improved interpretability and identified the most influential classification factors. The SHAP results showed that kinematic features were dominant, indicating that body-motion patterns are informative for detecting imbalance in VR. We also evaluated MI-CNN using only the top two-thirds of features ranked by SHAP importance. Despite a 33% reduction in input dimensionality, the model maintained performance, achieving 0.957 accuracy and 0.957 F1-score, with about a 1% decrease compared with the full-feature model. These findings suggest that multimodal sensing, temporal deep learning, and explainable AI can support reliable classification of balance-related instability in VR. Accurate recognition of imbalanced postural states may raise awareness of fall risk and guide safer, adaptive VR systems that respond to instability while improving user safety and experience. Code is available at: https://github.com/NipaAnjum/MI-CNN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。