基于原型对比学习，精准识别多标签教育视频中的细粒度内容
Class Prototypes based Contrastive Learning for Classifying Multi-Label and Fine-Grained Educational Videos
- 用类别原型构建对比损失，提升多标签细粒度分类性能
- 在193小时标注数据上达到超越基线的准确率，跨数据集表现稳健
- 适合教育内容筛选、儿童媒体安全评估等实际场景
儿童早期在线媒体消费增长促使教育者需要数据驱动工具来筛选适龄教育内容。本文聚焦识别人类语言与数学两类常见教育内容，依据美国共同核心标准选取子类别（如'字母名称'、'字母发音'、'数数'、'分类'）。由于视频可能包含多种内容且视觉相似（如'字母名称'与'字母发音'），问题被建模为细粒度多标签分类。提出基于类别原型的监督对比学习方法：为每类学习一个原型，通过损失函数最小化同类样本与原型距离，最大化异类样本距离。考虑到视听协同对理解至关重要，采用多模态Transformer捕捉视频中视觉与音频的交互。构建新数据集APPROVE，包含193小时专家标注视频，共19个细粒度教育类别。所提方法在APPROVE及Youtube-8M、COIN等基准上均优于强基线，数据集已公开于https://github.com/rohit-gupta/MMContrast/tree/main/APPROVE。
原文摘要 · Abstract (English)
The recent growth in the consumption of online media by children during early childhood necessitates data-driven tools enabling educators to filter out appropriate educational content for young learners. This paper presents an approach for detecting educational content in online videos. We focus on two widely used educational content classes: literacy and math. For each class, we choose prominent codes (sub-classes) based on the Common Core Standards. For example, literacy codes include `letter names', `letter sounds', and math codes include `counting', `sorting'. We pose this as a fine-grained multilabel classification problem as videos can contain multiple types of educational content and the content classes can get visually similar (e.g., `letter names' vs `letter sounds'). We propose a novel class prototypes based supervised contrastive learning approach that can handle fine-grained samples associated with multiple labels. We learn a class prototype for each class and a loss function is employed to minimize the distances between a class prototype and the samples from the class. Similarly, distances between a class prototype and the samples from other classes are maximized. As the alignment between visual and audio cues are crucial for effective comprehension, we consider a multimodal transformer network to capture the interaction between visual and audio cues in videos while learning the embedding for videos. For evaluation, we present a dataset, APPROVE, employing educational videos from YouTube labeled with fine-grained education classes by education researchers. APPROVE consists of 193 hours of expert-annotated videos with 19 classes. The proposed approach outperforms strong baselines on APPROVE and other benchmarks such as Youtube-8M, and COIN. The dataset is available at https://github.com/rohit-gupta/MMContrast/tree/main/APPROVE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。