提出新注意力机制，让摄像头测心率呼吸更准更稳
Efficient and Robust Multidimensional Attention in Remote Physiological Sensing through Target Signal Constrained Factorization
- 用生理信号特征约束注意力，精准提取视频中的生命体征
- 跨数据集测试中，心率与呼吸估计性能超越现有方法
- 模型轻量实时，适合在手机等设备上直接运行
基于摄像头的远程生理感应技术在医疗和人机交互领域具有变革潜力。尽管深度学习已提升从视频中提取生理信号的能力，但现有方法对域偏移的鲁棒性尚未充分评估。这些偏移包括光照变化、相机参数差异、头部运动、面部姿态及生理状态波动，显著影响实际表现。通过五个基准数据集的跨数据集评估，我们提出目标信号约束分解模块（TSFM），一种新型多维注意力机制，显式引入生理信号特性作为分解约束，实现更精确的特征提取。在此基础上，设计高效的双分支3D-CNN架构MMRPhys，可同时从多模态RGB与热成像视频中估计光体积描记图（rPPG）和呼吸信号（rRSP）。实验表明，搭载TSFM的MMRPhys在跨域泛化能力上显著优于当前最优方法，且推理延迟极低，适用于实时应用。本方法为鲁棒的多任务、多模态生理感知建立了新基准，并提供可部署于非受限环境的高效框架。基于浏览器的实时推理应用可在 https://physiologicailab.github.io/mmrphys-live 获取。
原文摘要 · Abstract (English)
Remote physiological sensing using camera-based technologies offers transformative potential for non-invasive vital sign monitoring across healthcare and human-computer interaction domains. Although deep learning approaches have advanced the extraction of physiological signals from video data, existing methods have not been sufficiently assessed for their robustness to domain shifts. These shifts in remote physiological sensing include variations in ambient conditions, camera specifications, head movements, facial poses, and physiological states which often impact real-world performance significantly. Cross-dataset evaluation provides an objective measure to assess generalization capabilities across these domain shifts. We introduce Target Signal Constrained Factorization module (TSFM), a novel multidimensional attention mechanism that explicitly incorporates physiological signal characteristics as factorization constraints, allowing more precise feature extraction. Building on this innovation, we present MMRPhys, an efficient dual-branch 3D-CNN architecture designed for simultaneous multitask estimation of photoplethysmography (rPPG) and respiratory (rRSP) signals from multimodal RGB and thermal video inputs. Through comprehensive cross-dataset evaluation on five benchmark datasets, we demonstrate that MMRPhys with TSFM significantly outperforms state-of-the-art methods in generalization across domain shifts for rPPG and rRSP estimation, while maintaining a minimal inference latency suitable for real-time applications. Our approach establishes new benchmarks for robust multitask and multimodal physiological sensing and offers a computationally efficient framework for practical deployment in unconstrained environments. The web browser-based application featuring on-device real-time inference of MMRPhys model is available at https://physiologicailab.github.io/mmrphys-live
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。