arXiv:2507.18675cs.CVcs.LG2025-07

用CLIP模型提升动作识别泛化能力,解决视觉线索丢失导致的误判问题。

Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

  • 测试三种遮蔽策略,发现关键视觉线索被遮后模型表现不稳定
  • 在UCF-101数据集上,遮蔽50%图像时准确率下降超20%
  • 引入类特定噪声增强注意力,提升准确率与模型置信度

人体动作识别在医疗领域至关重要,支持患者行为监测、跌倒检测、手术机器人监管和操作技能评估。传统模型如CNN和RNN虽取得一定成效,但在复杂多变的动作识别中泛化能力有限。基于Transformer的视觉语言模型CLIP为从视频数据中实现动作识别泛化提供了新可能。本文在UCF-101数据集上评估CLIP性能,系统分析三种遮蔽策略:(1)按比例和形状的黑色遮蔽(10%、30%、50%),(2)针对特征的遮蔽以抑制偏差诱导元素,(3)仅保留类别特异性区域的隔离遮蔽。结果表明,当关键视觉线索被遮挡时,CLIP表现出不一致行为及频繁误判。为此,我们提出通过自定义损失函数学习类特定噪声,强化模型对类别定义特征的关注。该方法提升了分类准确率与模型置信度,同时降低偏差。最后讨论了该模型在临床场景应用中的挑战,并提出未来工作方向,以提升在跨域独立医疗场景下的泛化能力。

原文摘要 · Abstract (English)

Human action recognition plays a critical role in healthcare and medicine, supporting applications such as patient behavior monitoring, fall detection, surgical robot supervision, and procedural skill assessment. While traditional models like CNNs and RNNs have achieved moderate success, they often struggle to generalize across diverse and complex actions. Recent advancements in vision-language models, especially the transformer-based CLIP model, offer promising capabilities for generalizing action recognition from video data. In this work, we evaluate CLIP on the UCF-101 dataset and systematically analyze its performance under three masking strategies: (1) percentage-based and shape-based black masking at 10%, 30%, and 50%, (2) feature-specific masking to suppress bias-inducing elements, and (3) isolation masking that retains only class-specific regions. Our results reveal that CLIP exhibits inconsistent behavior and frequent misclassifications, particularly when essential visual cues are obscured. To overcome these limitations, we propose incorporating class-specific noise, learned via a custom loss function, to reinforce attention to class-defining features. This enhancement improves classification accuracy and model confidence while reducing bias. We conclude with a discussion on the challenges of applying such models in clinical domains and outline directions for future work to improve generalizability across domain-independent healthcare scenarios.

动作识别CLIP医疗应用泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。