arXiv:2502.00702cs.HCcs.NI2025-02被引 1

用视频流实时监测心跳,精度远超纯音视频方案。

CardioLive: Empowering Video Streaming with Online Cardiac Monitoring

  • 融合音视频流设计多模态网络,捕捉心率时序与频谱特征。
  • 在真实场景下实现1.79 BPM误差,优于单模态方案69%以上。
  • 可即插即用部署于Zoom/YouTube,平均吞吐达98-116 FPS。

在线心脏监测(OCM)为下一代视频流平台带来新可能,支持远程健康、情绪计算与深度伪造检测。然而视频流中的生理信息长期被忽视。本文提出CardioLive,首个集成于视频流平台的在线心脏监测系统。通过利用天然共存的音视频流,设计了首个音视频联合网络CardioNet,融合时序与频谱特征提取机制,在真实视频流条件下保持鲁棒性。为实现按需服务,将CardioLive实现为即插即用中间件,并解决帧率变化与音视频不同步等实际问题。大量实验表明,系统均方误差(MAE)仅1.79 BPM,较纯视频与纯音频方案分别降低69.2%和81.2%。在Zoom和YouTube中,平均吞吐量分别达115.97和98.16 FPS。该工作为视频流系统开辟了新应用方向,代码即将开源。

原文摘要 · Abstract (English)

Online Cardiac Monitoring (OCM) emerges as a compelling enhancement for the next-generation video streaming platforms. It enables various applications including remote health, online affective computing, and deepfake detection. Yet the physiological information encapsulated in the video streams has been long neglected. In this paper, we present the design and implementation of CardioLive, the first online cardiac monitoring system in video streaming platforms. We leverage the naturally co-existed video and audio streams and devise CardioNet, the first audio-visual network to learn the cardiac series. It incorporates multiple unique designs to extract temporal and spectral features, ensuring robust performance under realistic video streaming conditions. To enable the Service-On-Demand online cardiac monitoring, we implement CardioLive as a plug-and-play middleware service and develop systematic solutions to practical issues including changing FPS and unsynchronized streams. Extensive experiments have been done to demonstrate the effectiveness of our system. We achieve a Mean Square Error (MAE) of 1.79 BPM error, outperforming the video-only and audio-only solutions by 69.2% and 81.2%, respectively. Our CardioLive service achieves average throughputs of 115.97 and 98.16 FPS when implemented in Zoom and YouTube. We believe our work opens up new applications for video stream systems. We will release the code soon.

心率监测音视频融合流媒体实时系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。