构建端到端的视觉语言模型，实现像素级跨帧统一理解。
From Pixels to Words -- Towards Native One-Vision Models at Scale
- 摒弃模块化设计，直接端到端学习像素与词语对应关系。
- 在视频理解任务上接近传统模型性能，且更擅长细粒度视觉感知。
- 适合研究多模态融合与大规模视觉语言建模的开发者参考。
当前视觉语言模型通常通过多阶段对齐拼接独立的图像编码器和语言解码器，这种模块化框架不可避免地割裂了像素级信号，并分散了早期像素-词语交互。与此同时，尽管原生视觉语言模型在单图任务上表现优异，但在多图、视频理解及空间智能方面仍缺乏探索。为此，我们提出NEO-ov，一种无需外部编码器、辅助适配器或后期融合的原生基础模型，可端到端学习跨帧与像素-词语对应关系。通过彻底消除模块边界，NEO-ov实现了原生的细粒度统一时空建模。值得注意的是，NEO-ov显著缩小了与模块化模型的差距，同时在细粒度视觉感知上表现更优，验证了原生‘一视觉’架构在规模化下的可行性与竞争力。此外，我们还提供了系统的架构分析与详细的训练方案，以推动后续原生多模态建模研究。代码与模型已公开于：https://github.com/EvolvingLMMs-Lab/NEO。
原文摘要 · Abstract (English)
Current vision-language models (VLMs) typically stitch together separate image encoders and language decoders via multi-stage alignment, a modular framework that inevitably fragments pixel-level signals across frames and scatters early pixel-word interactions. In parallel, native VLMs, despite impressive performance on single images, remain largely unexplored in multi-image, video understanding, and spatial intelligence. Hence, we introduce NEO-ov, a native foundation model that learns cross-frame and pixel-word correspondence end-to-end, without any external encoders, auxiliary adapters, or post-hoc fusion. By eliminating module boundaries entirely, NEO-ov enables fine-grained and unified spatiotemporal modeling to emerge natively inside the model. Notably, NEO-ov largely narrows the gap to modular counterparts while excelling at fine-grained visual perception, validating that native "one-vision" architectures are not only feasible but competitive at scale. Beyond empirical performance, we unveil systematic architectural analyses and detailed training recipes to facilitate subsequent native multimodal modeling. Our code and models are publicly available at: https://github.com/EvolvingLMMs-Lab/NEO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。