让视觉模型像视频编码器一样只关注重要信息,提升效率与准确率。
OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
- 采用码流对齐的稀疏化策略,仅处理3.1%-25%高信息量区域。
- 在16项任务中优于主流视觉模型,视频理解平均提升4.1%。
- 适合追求高效视觉通用模型的研究者和开发者。
人工智能通用性本质上是压缩问题。有效压缩需实现架构与数据结构的共振:当前视觉模型均匀处理密集像素,浪费算力于静态背景,忽视定义运动与意义的预测残差。本文提出OneVision-Encoder,通过码流对齐的块级稀疏化,仅计算3.1%-25%信号熵高的区域。为统一不规则令牌布局下的时空推理,该模型采用共享3D RoPE,并基于超百万语义概念的聚类判别目标训练,同时捕捉物体恒常性与运动动态。实验验证核心假设:效率与精度正相关。集成至LLM后,在16个图像、视频及文档理解基准上持续超越Qwen3-ViT与SigLIP2,尽管使用更少视觉令牌与预训练数据。尤其在视频理解任务中,平均性能提升4.1%。码流对齐的块级稀疏化是下一代视觉通用模型的基础原则。
原文摘要 · Abstract (English)
Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs. Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics. Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。