用改进的伪标签和时间一致性约束,让少量标注视频也能实现高精度动作检测。
Stable Mean Teacher for Semi-supervised Video Action Detection

- 设计错误恢复模块,让教师模型从学生错误中学习并优化未标注数据的伪标签。
- 在UCF101-24上仅用10%标签就达到监督模型100%标签的效果,平均提升23.5%。
- 方法通用性强,可扩展至大尺度数据集和视频分割任务,适合视频分析研究者。
本文研究视频动作检测中的半监督学习问题。视频动作检测需同时完成时空定位与分类,但标注数据有限易导致预测不可靠。提出Stable Mean Teacher框架,通过新设计的错误恢复(EoR)模块,让教师模型从学生在有标签样本上的错误中学习,并提升无标签样本的伪标签质量。针对现有时空损失忽略时间连贯性的问题,提出像素差分(DoP)约束,强化时间一致性,实现更稳定的时序检测。在四个基准数据集(UCF101-24、JHMDB21、AVA、YouTube-VOS)上评估,该方法在UCF101-24上比监督基线平均提升23.5%,在JHMDB21上提升16%,在AVA上提升3.3%。仅使用10%和20%标注数据时,在UCF101-24和JHMDB21上表现媲美全量标注的监督模型。进一步验证其在大规模数据集和视频对象分割任务上的泛化能力。代码与模型已开源。
原文摘要 · Abstract (English)
In this work, we focus on semi-supervised learning for video action detection. Video action detection requires spatiotemporal localization in addition to classification, and a limited amount of labels makes the model prone to unreliable predictions. We present Stable Mean Teacher, a simple end-to-end teacher-based framework that benefits from improved and temporally consistent pseudo labels. It relies on a novel Error Recovery (EoR) module, which learns from students' mistakes on labeled samples and transfers this knowledge to the teacher to improve pseudo labels for unlabeled samples. Moreover, existing spatiotemporal losses do not take temporal coherency into account and are prone to temporal inconsistencies. To address this, we present Difference of Pixels (DoP), a simple and novel constraint focused on temporal consistency, leading to coherent temporal detections. We evaluate our approach on four different spatiotemporal detection benchmarks: UCF101-24, JHMDB21, AVA, and YouTube-VOS. Our approach outperforms the supervised baselines for action detection by an average margin of 23.5% on UCF101-24, 16% on JHMDB21, and 3.3% on AVA. Using merely 10% and 20% of data, it provides competitive performance compared to the supervised baseline trained on 100% annotations on UCF101-24 and JHMDB21, respectively. We further evaluate its effectiveness on AVA for scaling to large-scale datasets and YouTube-VOS for video object segmentation, demonstrating its generalization capability to other tasks in the video domain. Code and models are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。