用帧聚合提升视频表征,准确率超现有方法且更省算力
Supervised Contrastive Frame Aggregation for Video Representation Learning
- 将多帧视频拼成图像输入,兼容预训练卷积模型
- 在Penn Action达76%准确率,HMDB51达48%,显著优于ViVIT
- 无需复杂Transformer,适合资源有限场景
我们提出一种基于监督对比学习的视频表征学习框架,利用时间全局上下文。引入视频到图像的帧聚合策略,将每段视频的多个帧空间排列为单张输入图像,可直接使用ResNet50等预训练卷积网络骨干,避免复杂视频Transformer带来的计算开销。设计了一种对比学习目标,直接比较模型生成的成对投影:同标签视频的投影作为正样本,其余均为负样本。通过同一视频的不同时间采样方式生成多个自然视角,无需依赖数据增强即可产生多样化的正样本,增强全局上下文并减少过拟合。在Penn Action和HMDB51数据集上的实验表明,该方法在分类准确率上优于现有方法,同时所需计算资源更少。所提方法在监督与自监督设置下均能学习有效视频表征,支持分类、字幕生成等任务。在Penn Action上达到76%准确率,远高于ViVIT的43%;在HMDB51上达48%,优于ViVIT的37%。
原文摘要 · Abstract (English)
We propose a supervised contrastive learning framework for video representation learning that leverages temporally global context. We introduce a video to image aggregation strategy that spatially arranges multiple frames from each video into a single input image. This design enables the use of pre trained convolutional neural network backbones such as ResNet50 and avoids the computational overhead of complex video transformer models. We then design a contrastive learning objective that directly compares pairwise projections generated by the model. Positive pairs are defined as projections from videos sharing the same label while all other projections are treated as negatives. Multiple natural views of the same video are created using different temporal frame samplings from the same underlying video. Rather than relying on data augmentation these frame level variations produce diverse positive samples with global context and reduce overfitting. Experiments on the Penn Action and HMDB51 datasets demonstrate that the proposed method outperforms existing approaches in classification accuracy while requiring fewer computational resources. The proposed Supervised Contrastive Frame Aggregation method learns effective video representations in both supervised and self supervised settings and supports video based tasks such as classification and captioning. The method achieves seventy six percent classification accuracy on Penn Action compared to forty three percent achieved by ViVIT and forty eight percent accuracy on HMDB51 compared to thirty seven percent achieved by ViVIT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。