ELVA无需视觉编码器，实现高效视频语言理解。
Breaking the Encoder Barrier for Seamless Video-Language Understanding
- 摒弃传统编码器，直接建模视频与文本的细粒度交互
- 仅用700万数据对，性能媲美主流模型，计算量降低95%
- 适合追求实时性与低资源部署的视频理解场景
大多数视频-大语言模型采用编码器-解码器架构，由视觉编码器提取帧级特征供语言模型处理。但该方法计算开销大、存在分辨率偏差，且难以捕捉细粒度多模态交互。为此，我们提出无编码器视频大模型ELVA，直接建模精细的视频-语言交互，无需依赖视觉编码器。ELVA采用标记合并构建自底向上的层次化表示，并引入视频引导监督器实现时空表征的直接学习。此外，混合分辨率机制策略性地融合高/低分辨率帧作为输入，在性能与效率间取得最佳平衡。仅使用700万公开视频-文本对，ELVA性能达到与编码器型模型相当水平，同时将浮点运算量减少高达95%，推理延迟降低92%，为实时视频理解提供可扩展、高效的解决方案。
原文摘要 · Abstract (English)
Most Video-Large Language Models (Video-LLMs) adopt an encoder-decoder framework, where a vision encoder extracts frame-wise features for processing by a language model. However, this approach incurs high computational costs, introduces resolution biases, and struggles to capture fine-grained multimodal interactions. To overcome these limitations, we propose ELVA, an encoder-free Video-LLM that directly models nuanced video-language interactions without relying on a vision encoder. ELVA employs token merging to construct a bottom-up hierarchical representation and incorporates a video guidance supervisor for direct spatiotemporal representation learning. Additionally, a hybrid-resolution mechanism strategically integrates high- and low-resolution frames as inputs to achieve an optimal balance between performance and efficiency. With only 7M publicly available video-text pairs, ELVA achieves performance on par with encoder-based Video-LLMs while reducing FLOPs by up to 95\% and inference latency by 92\%, offering a scalable and efficient solution for real-time video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。