arXiv:2607.24904cs.CVcs.CL2026-07被引 3

让视觉语言模型更快理解视频,效率提升3.5倍。

Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

论文配图:Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model
图 1 · 摘自论文原文
  • 用运动矢量和残差能量只编码动态关键区域,减少75%视觉令牌
  • 在静态任务上媲美大模型,视频理解与空间推理显著更强
  • 适合实时视频分析、机器人感知等需要快速响应的场景

标准视觉语言模型存在莫拉维克悖论:擅长复杂离线视觉推理,却在简单流式感知任务上表现差且低效。我们提出Mage-VL,一种高效编码器原生流式多模态基础模型,支持实时多模态理解与交互。其核心是自研分词器Mage-ViT,通过运动矢量与残差能量,在16×16块级别选择性编码动态、高熵区域,取代均匀帧采样,将视觉令牌消耗降低超75%,同时保留时空上下文。在约5.6亿未标注图像和1亿未标注视频帧上从头训练,性能达到或超越基于数十亿图文对训练的旗舰编码器。我们建立AI4AI数据流水线,涵盖提示-代码联合优化与AI驱动性能诊断,指导训练策略。通过仿生双系统架构——轻量级系统1事件门控与因果系统2解码器,实现主动流式感知。大量评估显示,Mage-VL-4B在静态任务上匹配Qwen3-VL-4B,视频理解与2D/3D空间推理显著提升,壁钟推理速度最高加快3.5倍,全面超越150亿参数的Phi-4-reasoning-vision基线。此外,我们发现七个关键经验:预训练数据效率、可变分辨率扩展、编码器系统加速、VideoQA微调冗余、运动-空间协同、AI4AI数据流水线、多模态强化学习零视觉微调。

原文摘要 · Abstract (English)

Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.

多模态模型视频理解流式处理编码器优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。