轻量模型+时空动作线索，实现高效视频显著性预测
Minimalistic Video Saliency Prediction via Efficient Decoder & Spatio Temporal Action Cues
- 采用轻量化解码器与时空动作线索融合，压缩模型至36MB
- 在9个数据集上超越主流模型，36MB版本达1000fps以上
- 适合低资源部署，尤其适用于实时视频分析场景
本文提出基于ViNet架构的轻量级模型ViNet-S（36MB），采用U-Net结构设计的轻量化解码器，在不损失性能的前提下显著减小模型规模。此外，ViNet-A（148MB）引入时空动作定位（STAL）特征，区别于传统使用动作分类主干的视频显著性模型。实验表明，通过平均两个模型的显著性图生成结果，该集成方法在三个纯视觉和六个视听显著性数据集上达到当前最优性能，且在参数效率与实时性方面优于基于Transformer的模型，其中ViNet-S推理速度超过1000fps。
原文摘要 · Abstract (English)
This paper introduces ViNet-S, a 36MB model based on the ViNet architecture with a U-Net design, featuring a lightweight decoder that significantly reduces model size and parameters without compromising performance. Additionally, ViNet-A (148MB) incorporates spatio-temporal action localization (STAL) features, differing from traditional video saliency models that use action classification backbones. Our studies show that an ensemble of ViNet-S and ViNet-A, by averaging predicted saliency maps, achieves state-of-the-art performance on three visual-only and six audio-visual saliency datasets, outperforming transformer-based models in both parameter efficiency and real-time performance, with ViNet-S reaching over 1000fps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。