通过遮蔽背景和物体，减少模型对静态特征的依赖，提升零样本动作识别效果。
Can masking background and object reduce static bias for zero-shot action recognition?
- 训练时遮蔽背景、物体或人物，测试模型对动作的注意力
- 遮蔽背景导致Kinetics400上性能下降，表明模型依赖背景信息
- 在弱背景偏差数据集上遮蔽背景反而提升性能，适合关注动作本质的研究者
本文研究零样本动作识别中的静态偏差问题。尽管基于CLIP的模型广泛使用，但其是否真正聚焦于人体动作仍不明确，因CLIP主要捕捉与语言相关的外观特征。本研究通过在训练和验证阶段分别遮蔽背景、物体和人物，探究静态偏差的影响。实验显示，遮蔽背景会使Kinetics400上的性能下降，表明模型依赖背景；而在Mimetics（弱背景偏差）上，遮蔽背景反而提升性能。此外，在具有强物体偏差的SSv2上，同时遮蔽背景和物体（不同颜色）可提高性能。结果表明，训练时遮蔽静态元素能有效抑制模型对静态特征的依赖，使其更关注人类动作本身。
原文摘要 · Abstract (English)
In this paper, we address the issue of static bias in zero-shot action recognition. Action recognition models need to represent the action itself, not the appearance. However, some fully-supervised works show that models often rely on static appearances, such as the background and objects, rather than human actions. This issue, known as static bias, has not been investigated for zero-shot. Although CLIP-based zero-shot models are now common, it remains unclear if they sufficiently focus on human actions, as CLIP primarily captures appearance features related to languages. In this paper, we investigate the influence of static bias in zero-shot action recognition with CLIP-based models. Our approach involves masking backgrounds, objects, and people differently during training and validation. Experiments with masking background show that models depend on background bias as their performance decreases for Kinetics400. However, for Mimetics, which has a weak background bias, masking the background leads to improved performance even if the background is masked during validation. Furthermore, masking both the background and objects in different colors improves performance for SSv2, which has a strong object bias. These results suggest that masking the background or objects during training prevents models from overly depending on static bias and makes them focus more on human action.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。