仅用视频实现高精度实时晕动症预测,适合消费级VR部署
Towards Consumer-Grade Cybersickness Prediction: Multi-Model Alignment for Real-Time Vision-Only Inference
- 用多模态图神经网络融合头姿、眼动等非侵入信号
- 88.4%预测准确率接近脑电方法,延迟仅90ms
- 可纯视频推理,适合普通VR设备实时应用
晕动症仍是沉浸式虚拟现实(VR)在消费级场景中普及的主要障碍。以往方法依赖脑电图(EEG)等侵入性信号以获得高预测精度,但需专用硬件,难以实际应用。本文提出一种可扩展、可部署的个性化晕动症预测框架,仅使用商业VR头显中可用的非侵入信号(如头部运动、眼动追踪和生理反应)。模型采用模态特定的图神经网络,并引入差异注意力模块,提取跨模态动态变化的时空嵌入;通过跨模态对齐模块联合训练视频编码器,使其学习个体特征,从而在推理阶段仅用视频输入即可精准预测晕动症。实验表明,该模型达到88.4%的准确率,接近基于EEG的方法(89.16%),同时将部署复杂度大幅降低。平均推理延迟为90ms,支持实时应用,适用于集成至消费级VR平台而不牺牲个性化与性能。代码将发布于 https://github.com/U235-Aurora/PTGNN。
原文摘要 · Abstract (English)
Cybersickness remains a major obstacle to the widespread adoption of immersive virtual reality (VR), particularly in consumer-grade environments. While prior methods rely on invasive signals such as electroencephalography (EEG) for high predictive accuracy, these approaches require specialized hardware and are impractical for real-world applications. In this work, we propose a scalable, deployable framework for personalized cybersickness prediction leveraging only non-invasive signals readily available from commercial VR headsets, including head motion, eye tracking, and physiological responses. Our model employs a modality-specific graph neural network enhanced with a Difference Attention Module to extract temporal-spatial embeddings capturing dynamic changes across modalities. A cross-modal alignment module jointly trains the video encoder to learn personalized traits by aligning video features with sensor-derived representations. Consequently, the model accurately predicts individual cybersickness using only video input during inference. Experimental results show our model achieves 88.4\% accuracy, closely matching EEG-based approaches (89.16\%), while reducing deployment complexity. With an average inference latency of 90ms, our framework supports real-time applications, ideal for integration into consumer-grade VR platforms without compromising personalization or performance. The code will be relesed at https://github.com/U235-Aurora/PTGNN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。