用热成像指导相机采样,只处理关键手物交互画面,省电又高效。
THOR: Thermal-guided Hand-Object Reasoning via Adaptive Vision Sampling
- 热成像检测手部动作切换,动态调节可见光帧率
- 仅用3%原始视频数据就捕捉全部活动片段
- 适合长期监测手部行为的可穿戴设备应用
可穿戴摄像头在观察和干预人类行为方面日益重要,尤其在记录手部活动时提供详细视觉数据。然而,持续处理RGB图像会显著耗电、产生大量冗余视频数据、引发隐私问题,并需大量算力支持实时分析。我们提出THOR,一种基于热成像的自适应时空帧采样方法,通过低分辨率热成像识别手部活动切换时刻,动态提升活动转换期的帧率,降低持续活动期的采样频率。同时,利用热信号定位每帧中手-物体交互区域,仅裁剪并处理必要图像部分以进行活动识别。我们在14名参与者的真实场景中验证该系统,并在包含923名参与者、覆盖9个国家、总计3670小时视频的Ego4D数据集上评估。结果显示,仅使用3%原始RGB视频数据,即可完整捕获所有活动片段,且手部活动识别F1分数达95%,与使用完整视频的94%相当。本工作为长期实时监测手部活动及健康风险行为提供了更实用的路径。
原文摘要 · Abstract (English)
Wearable cameras are increasingly used as an observational and interventional tool for human behaviors by providing detailed visual data of hand-related activities. This data can be leveraged to facilitate memory recall for logging of behavior or timely interventions aimed at improving health. However, continuous processing of RGB images from these cameras consumes significant power impacting battery lifetime, generates a large volume of unnecessary video data for post-processing, raises privacy concerns, and requires substantial computational resources for real-time analysis. We introduce THOR, a real-time adaptive spatio-temporal RGB frame sampling method that leverages thermal sensing to capture hand-object patches and classify them in real-time. We use low-resolution thermal camera data to identify moments when a person switches from one hand-related activity to another, and adjust the RGB frame sampling rate by increasing it during activity transitions and reducing it during periods of sustained activity. Additionally, we use the thermal cues from the hand to localize the region of interest (i.e., the hand-object interaction) in each RGB frame, allowing the system to crop and process only the necessary part of the image for activity recognition. We develop a wearable device to validate our method through an in-the-wild study with 14 participants and over 30 activities, and further evaluate it on Ego4D (923 participants across 9 countries, totaling 3,670 hours of video). Our results show that using only 3% of the original RGB video data, our method captures all the activity segments, and achieves hand-related activity recognition F1-score (95%) comparable to using the entire RGB video (94%). Our work provides a more practical path for the longitudinal use of wearable cameras to monitor hand-related activities and health-risk behaviors in real time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。