通过多视角动作与文本描述,提升零样本动作识别在不同视角下的泛化能力。
Cross-Domain Human Action Recognition from Multiview Motion and Textual Descriptions

- 融合多视角运动信息与文本描述,构建面向姿态的特征编码
- 在多个数据集上超越现有零样本方法,尤其在视角变化下表现优异
- 适合需要跨域泛化、少标注场景的动作识别应用
真实场景中动作识别系统需应对域偏移甚至未见动作的挑战。当前零样本动作识别(ZSAR)方法通常假设测试时视角与训练一致,但人体朝向和相机视角变化会引入显著域差距,限制模型泛化能力。本文提出一种新的姿态感知动作识别方法,结合多视角运动信号与动作文本描述,在训练阶段学习不同姿态下的运动特征,并在推理时使用对应姿态的文本提示进行匹配。大量实验表明,该方法在NTU-RGB+D、BABEL、NW-UCLA及两个监控数据集上均持续优于最新零样本方法。所学表征具备强迁移能力,在跨域及同域已见动作识别任务中均表现优异。代码与模型已公开于https://icb-vision-ai.github.io/OrientationAware-HAR。
原文摘要 · Abstract (English)
Robustness to domain changes is a key capability for effective deployment of human action recognition systems in real-world scenarios, where action categories at inference can present important domain shifts or even unseen actions from training. In this context, improving the recognition capabilities of Zero-Shot Action Recognition models (ZSAR), without requiring strong annotation efforts, remains a central challenge. Most ZSAR approaches assume that actions are observed under geometric conditions similar to those seen during training. In practice, variations in human body orientation and camera viewpoint add a significant domain gap in ZSAR, substantially limiting generalization to novel action-motion combinations. In this context, this paper presents a novel orientation-aware action recognition approach with improved cross-domain capabilities. Our approach combines motion cues of multiple camera viewpoints and text descriptions of human actions in the training phase. We present a new orientation-aware motion encoding network to learn different motion features, and adapt a specific orientation-aware text prompt to match the corresponding features at inference. Extensive experiments demonstrate that the proposed method consistently improves ZSAR performance across different recognition benchmarks, outperforming recent state-of-the-art zero-shot approaches on NTU-RGB+D, BABEL, NW-UCLA, and on two surveillance datasets. In addition, the learned representations exhibit strong transfer learning capabilities, yielding competitive performance on both cross-domain and same-domain recognition of seen actions. Code and trained models are available at: https://icb-vision-ai.github.io/OrientationAware-HAR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。