arXiv:2503.07259cs.CVcs.AI2025-03被引 10

用视频知识教IMU传感器识别动作,省电又隐私。

COMODO: Cross-Modal Video-to-IMU Distillation for Efficient Egocentric Human Activity Recognition

  • 用预训练视频模型指导IMU特征学习,无须标签。
  • 在多个数据集上性能达监督模型水平,跨数据集泛化强。
  • 适合做低功耗、持续运行的可穿戴动作识别系统。

构建面向持续活动理解的人类中心可穿戴系统面临根本矛盾:基于第一视角视频的模型虽具丰富语义信息且性能优越,但功耗高、隐私风险大且依赖光照,难以实现持续本地推理;而惯性测量单元(IMU)传感器能耗低、保护隐私,却缺乏大规模标注数据,泛化能力弱。为此,本文提出COMODO——一种跨模态自监督蒸馏框架,无需标签即可将视频中的语义知识迁移到IMU。COMODO利用预训练并冻结的视频编码器构建动态实例队列,对齐视频与IMU嵌入特征分布,使IMU编码器继承视频的丰富语义结构,同时保持高效性。在多个第一视角动作识别数据集上的实验表明,COMODO显著提升下游性能,达到甚至超越全监督模型水平,并展现出优异的跨数据集泛化能力。其简单灵活的设计兼容多种预训练视频与时序模型,为未来普适计算研究中使用更强大的师生基础模型提供可能。代码已开源:https://github.com/cruiseresearchgroup/COMODO。

原文摘要 · Abstract (English)

The goal of creating intelligent, human-centered wearable systems for continuous activity understanding faces a fundamental trade-off: Egocentric video-based models capture rich semantic information and have demonstrated strong performance in human activity recognition (HAR), but their high power consumption, privacy concerns, and dependence on lighting limit their feasibility for continuous on-device recognition. In contrast, inertial measurement unit (IMU) sensors offer an energy-efficient, privacy-preserving alternative, yet lack large-scale annotated datasets, leading to weaker generalization. To bridge this gap, we propose COMODO, a cross-modal self-supervised distillation framework that transfers semantic knowledge from video to IMU without requiring labels. COMODO leverages a pretrained and frozen video encoder to construct a dynamic instance queue to align the feature distributions of video and IMU embeddings. This enables the IMU encoder to inherit rich semantic structure from video while maintaining its efficiency for real-world applications. Experiments on multiple egocentric HAR datasets show that COMODO consistently improves downstream performance, matching or surpassing fully supervised models, and demonstrating strong cross-dataset generalization. Benefiting from its simplicity and flexibility, COMODO is compatible with diverse pretrained video and time-series models, offering the potential to leverage more powerful teacher and student foundation models in future ubiquitous computing research. The code is available at this repository: https://github.com/cruiseresearchgroup/COMODO.

动作识别跨模态自监督可穿戴

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。