用32个令牌高效表示视频，性能媲美大模型
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs
- 引入时序编码器压缩多帧视频为32个视觉令牌
- 40亿参数模型在视频问答上达到340亿模型水平
- 适合需要轻量高效视频理解的场景
我们提出xGen-MM-Vid（BLIP-3-Video）：一种专为视频设计的多模态语言模型，能高效捕捉多帧间的时序信息。该模型在传统视觉分词器基础上引入时序编码器，将多帧序列映射为紧凑的视觉令牌集合，显著减少所需视觉令牌数（如32对比4608）。我们探索了可学习的空间-时序池化及序列模型（如Token Turing Machines）等时序编码方式。实验表明，尽管仅使用40亿参数且仅需32个视觉令牌，BLIP-3-Video在视频问答任务上的准确率仍可媲美更大规模的前沿模型（如340亿参数模型），兼具高效与高性能。
原文摘要 · Abstract (English)
We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames. BLIP-3-Video takes advantage of the 'temporal encoder' in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens. This enables BLIP3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs. 4608 tokens). We explore different types of temporal encoders, including learnable spatio-temporal pooling as well as sequential models like Token Turing Machines. We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using fewer visual tokens. The project website is at https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。