仅用1个标注句实现视频描述生成，通过关键词精修提升伪标签质量。
Pseudo-labeling with Keyword Refining for Few-Supervised Video Captioning
- 用动作引导的词级编辑生成伪标签，避免无效修改。
- 在MSR-VTT上仅用1个真标签时BLEU-4达29.6，超越基线。
- 适合标注成本高的视频描述场景，尤其适合小样本训练。
视频字幕生成旨在描述视频内容。现有方法通常需要每段视频提供10至20个标注字幕进行训练，成本高昂。本文探索仅使用一个或极少量真实字幕的可行性，提出少监督视频字幕任务。我们设计了一种包含词汇约束伪标签模块与关键词精修字幕模块的框架。前者利用预训练的词级分类器，通过复制、替换、插入、删除等动作引导模型编辑词汇，并借助预训练语言模型微调候选句子；同时采用重复惩罚采样策略，鼓励生成简洁无冗余的伪标签，并基于预训练视频-文本模型筛选最相关句子。为保证伪标签与视频内容语义一致，我们引入基于Transformer的关键词精修器，结合视频-关键词门控融合机制强化相关词汇。在多个基准数据集上的实验表明，该方法在少监督和全监督场景下均表现优异。代码已开源：https://github.com/mlvccn/PKG_VidCap。
原文摘要 · Abstract (English)
Video captioning generate a sentence that describes the video content. Existing methods always require a number of captions (\eg, 10 or 20) per video to train the model, which is quite costly. In this work, we explore the possibility of using only one or very few ground-truth sentences, and introduce a new task named few-supervised video captioning. Specifically, we propose a few-supervised video captioning framework that consists of lexically constrained pseudo-labeling module and keyword-refined captioning module. Unlike the random sampling in natural language processing that may cause invalid modifications (\ie, edit words), the former module guides the model to edit words using some actions (\eg, copy, replace, insert, and delete) by a pretrained token-level classifier, and then fine-tunes candidate sentences by a pretrained language model. Meanwhile, the former employs the repetition penalized sampling to encourage the model to yield concise pseudo-labeled sentences with less repetition, and selects the most relevant sentences upon a pretrained video-text model. Moreover, to keep semantic consistency between pseudo-labeled sentences and video content, we develop the transformer-based keyword refiner with the video-keyword gated fusion strategy to emphasize more on relevant words. Extensive experiments on several benchmarks demonstrate the advantages of the proposed approach in both few-supervised and fully-supervised scenarios. The code implementation is available at https://github.com/mlvccn/PKG_VidCap
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。