用视觉语言模型生成伪标签,实现零标注视频预训练。
LEViL: Label-Efficient Video Learning via Zero-Shot Distillation over VLM-Generated Pseudo-Label Spaces

- 用VLM自动生成视频文本描述,构建可解释的伪标签空间。
- 在有限标注下优于现有半监督方法,全数据微调也表现优异。
- 适合无标注数据丰富的视频任务,尤其适用于跨域迁移场景。
监督视频预训练是提升下游动作识别性能的常用迁移学习方法,但需大规模标注源数据,且性能受源域与目标域相似性影响。构建不同目标域的标注预训练数据集成本高、难扩展。为此,本文提出一种标签高效视频学习框架,结合无标注视频预训练与目标标签集感知微调。预训练阶段,利用视觉语言模型(VLM)生成未标注视频的文本描述,并构建可解释的语义伪标签空间;冻结的视频-语言模型在该空间上生成零样本软目标分布,使学生视频编码器无需人工标注即可学习语义丰富表征。下游适应阶段,目标标签集感知微调融合有标注目标视频的监督学习与实际目标标签集上的零样本蒸馏,兼顾语义指导与任务适配。在UCF101和HMDB51上的实验表明,该框架在所有有限标注设置下均优于对比的半监督视频动作识别方法。此外,无标注预训练阶段学习到的可迁移表征,即使基于相对较小的无标注预训练池,也能为全数据微调提供有效初始化。
原文摘要 · Abstract (English)
Supervised video pretraining is a common transfer learning practice for improving downstream action recognition performance. However, it requires large-scale labeled source datasets, and the effectiveness of the learned initialization is influenced by the similarity between the source and target domains. Constructing such labeled pretraining datasets for different target domains is costly and difficult to scale. To address these limitations, this study proposes a label-efficient video learning framework that combines annotation-free video pretraining with target-label-set-aware fine-tuning. During pretraining, a vision-language model (VLM) generates textual descriptions of unlabeled videos, which are processed to construct an interpretable semantic pseudo-label space. A frozen video-language model then produces zero-shot soft target distributions over this space, allowing a student video encoder to learn semantically rich representations without manual source annotations. During downstream adaptation, target-label-set-aware fine-tuning combines supervised learning from labeled target videos with zero-shot distillation over the actual target label set, helping preserve VLM-derived semantic guidance while adapting the pretrained encoder to the target task. Experiments on UCF101 and HMDB51 show that the proposed framework outperforms the compared semi-supervised video action recognition methods across all evaluated limited-label regimes. Moreover, the annotation-free pretraining stage learns transferable representations that provide an effective initialization for full-data fine-tuning, despite relying on a comparatively modest unlabeled pretraining pool.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。