轻量化视频文本模型MobileViCLIP，可在手机上快速运行并精准检索。
MobileViCLIP: An Efficient Video-Text Model for Mobile Devices
- 将时序结构重参数化引入轻量图像文本模型，适配移动端推理。
- 移动端推理速度比InternVideo快55.4倍，零样本检索性能接近大模型。
- 适合移动设备部署，尤其适用于实时视频理解与跨模态搜索场景。
由于推理速度快且易于在移动端部署，高效轻量级神经网络日益受到关注。然而，现有视频预训练模型仍以高延迟的ViT架构为主，少有工作针对移动端构建高效架构。本文通过将时序结构重参数化引入高效图像文本模型，并在大规模高质量视频文本数据集上训练，提出可在移动端运行的高效视频文本模型MobileViCLIP，具备强大的零样本分类与检索能力。在移动端推理速度上，MobileViCLIP-Small 比 InternVideo2-L14 快 55.4 倍，比 InternVideo2-S14 快 6.7 倍；在 MSR-VTT 数据集上的零样本检索性能与 InternVideo2-L14 相当，优于 InternVideo2-S14 6.9%。代码已开源。
原文摘要 · Abstract (English)
Efficient lightweight neural networks are with increasing attention due to their faster reasoning speed and easier deployment on mobile devices. However, existing video pre-trained models still focus on the common ViT architecture with high latency, and few works attempt to build efficient architecture on mobile devices. This paper bridges this gap by introducing temporal structural reparameterization into an efficient image-text model and training it on a large-scale high-quality video-text dataset, resulting in an efficient video-text model that can run on mobile devices with strong zero-shot classification and retrieval capabilities, termed as MobileViCLIP. In particular, in terms of inference speed on mobile devices, our MobileViCLIP-Small is 55.4x times faster than InternVideo2-L14 and 6.7x faster than InternVideo2-S14. In terms of zero-shot retrieval performance, our MobileViCLIP-Small obtains similar performance as InternVideo2-L14 and obtains 6.9\% better than InternVideo2-S14 on MSR-VTT. The code is available at https://github.com/MCG-NJU/MobileViCLIP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。