用普通ResNet+大模型实现零样本视频理解，效果领先
ResNetVLLM -- Multi-modal Vision LLM for the Video Understanding Task
- 用未预训练的ResNet提取视觉特征，与大模型统一建模
- 在多个基准上达零样本视频理解最佳性能
- 适合想避开复杂视频模型、追求简洁高效的开发者
本文提出ResNetVLLM（基于ResNet的视觉大模型），一种新型跨模态框架，用于零样本视频理解。该框架将基于ResNet的视觉编码器与大型语言模型（LLM）结合，避免依赖预训练视频理解模型，转而使用未预训练的ResNet提取视觉特征。这种设计使模型在统一架构中学习视觉与语义表示，显著提升从视频输入生成准确、上下文相关的文本描述的能力。实验表明，ResNetVLLM在多个基准测试中表现卓越，包括MSRVTT-QA、MSVD-QA、TGIF-QA FrameQA和ActivityNet-QA，达到零样本视频理解（ZSVU）当前最优水平。
原文摘要 · Abstract (English)
In this paper, we introduce ResNetVLLM (ResNet Vision LLM), a novel cross-modal framework for zero-shot video understanding that integrates a ResNet-based visual encoder with a Large Language Model (LLM. ResNetVLLM addresses the challenges associated with zero-shot video models by avoiding reliance on pre-trained video understanding models and instead employing a non-pretrained ResNet to extract visual features. This design ensures the model learns visual and semantic representations within a unified architecture, enhancing its ability to generate accurate and contextually relevant textual descriptions from video inputs. Our experimental results demonstrate that ResNetVLLM achieves state-of-the-art performance in zero-shot video understanding (ZSVU) on several benchmarks, including MSRVTT-QA, MSVD-QA, TGIF-QA FrameQA, and ActivityNet-QA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。