用Mamba模型高效捕捉面部视频中的生理信号长时依赖。
PhysMamba: Efficient Remote Physiological Measurement with SlowFast Temporal Difference Mamba
- 引入时差Mamba模块增强局部动态差异，建模长时空上下文。
- 双流SlowFast结构融合多尺度时间特征，提升精度与效率。
- 在三个数据集上验证效果，适合无接触心率监测应用。
基于面部视频的远程光电容积脉搏波描记术（rPPG）旨在无需接触即可测量生理信号并监测心脏活动，在多种应用场景中展现出巨大潜力。以往基于深度学习的rPPG方法主要依赖于卷积神经网络（CNN）和变换器（Transformer）。然而，CNN的感知范围有限，难以捕捉长时序时空依赖；而变换器在处理长视频序列时也面临高复杂度问题。最近，状态空间模型（SSM）代表——Mamba，在长序列中捕捉长时依赖方面表现优异。本文提出PhysMamba，一种基于Mamba的框架，用于高效表征面部视频中的长时生理依赖。具体地，我们引入时差Mamba模块，先增强局部动态差异，再建模长时空上下文。此外，采用双流SlowFast架构融合多尺度时间特征。在三个基准数据集上的大量实验表明，PhysMamba具有优越性与高效性。代码已开源：https://github.com/Chaoqi31/PhysMamba。
原文摘要 · Abstract (English)
Facial-video based Remote photoplethysmography (rPPG) aims at measuring physiological signals and monitoring heart activity without any contact, showing significant potential in various applications. Previous deep learning based rPPG measurement are primarily based on CNNs and Transformers. However, the limited receptive fields of CNNs restrict their ability to capture long-range spatio-temporal dependencies, while Transformers also struggle with modeling long video sequences with high complexity. Recently, the state space models (SSMs) represented by Mamba are known for their impressive performance on capturing long-range dependencies from long sequences. In this paper, we propose the PhysMamba, a Mamba-based framework, to efficiently represent long-range physiological dependencies from facial videos. Specifically, we introduce the Temporal Difference Mamba block to first enhance local dynamic differences and further model the long-range spatio-temporal context. Moreover, a dual-stream SlowFast architecture is utilized to fuse the multi-scale temporal features. Extensive experiments are conducted on three benchmark datasets to demonstrate the superiority and efficiency of PhysMamba. The codes are available at https://github.com/Chaoqi31/PhysMamba
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。