提出五个新数据集，揭示遮挡对视频动作检测的影响及模型应对策略。
On Occlusions in Video Action Detection: Benchmark Datasets And Training Recipes
- 构建合成与真实遮挡数据集，系统评估模型在不同遮挡下的表现。
- 基于新训练方法，模型在遮挡场景下性能提升超32%，显著优于现有方法。
- 适合关注视频理解鲁棒性与遮挡建模的研究者和工程师。
本文研究视频动作检测中遮挡的影响。为此，我们提出了五个新的基准数据集：包含合成控制静态/动态遮挡的 O-UCF 与 O-JHMDB，包含真实运动遮挡的 OVIS-UCF 与 OVIS-JHMDB，以及模拟真实世界场景遮挡的 Real-OUCF。我们正式验证了直观预期：随着遮挡严重度增加，现有模型性能显著下降，且静态与动态遮挡下模型行为不同。我们发现神经网络中存在若干有趣现象：1）Transformer 模型天然优于 CNN，即使后者曾使用遮挡作为数据增强；2）在骨干网络中引入胶囊等符号组件，可使模型绑定训练中未见的遮挡物；3）在无实例级监督、蒸馏或对比学习目标下，真实图像/视频中仍能出现“共识岛屿”。这些涌现特性催生了简单有效的训练方案，使模型在诱导阶段（分组/分离）具备鲁棒性。采用该方案的模型在 O-UCF 上提升 32.3%，在 O-JHMDB 上提升 32.7%，在 Real-OUCF 上提升 2.6%（以 vMAP 衡量）。代码已开源。
原文摘要 · Abstract (English)
This paper explores the impact of occlusions in video action detection. We facilitate this study by introducing five new benchmark datasets namely O-UCF and O-JHMDB consisting of synthetically controlled static/dynamic occlusions, OVIS-UCF and OVIS-JHMDB consisting of occlusions with realistic motions and Real-OUCF for occlusions in realistic-world scenarios. We formally confirm an intuitive expectation: existing models suffer a lot as occlusion severity is increased and exhibit different behaviours when occluders are static vs when they are moving. We discover several intriguing phenomenon emerging in neural nets: 1) transformers can naturally outperform CNN models which might have even used occlusion as a form of data augmentation during training 2) incorporating symbolic-components like capsules to such backbones allows them to bind to occluders never even seen during training and 3) Islands of agreement can emerge in realistic images/videos without instance-level supervision, distillation or contrastive-based objectives2(eg. video-textual training). Such emergent properties allow us to derive simple yet effective training recipes which lead to robust occlusion models inductively satisfying the first two stages of the binding mechanism (grouping/segregation). Models leveraging these recipes outperform existing video action-detectors under occlusion by 32.3% on O-UCF, 32.7% on O-JHMDB & 2.6% on Real-OUCF in terms of the vMAP metric. The code for this work has been released at https://github.com/rajatmodi62/OccludedActionBenchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。