Oryx可按需处理任意分辨率和时长的视觉数据,兼顾效率与精度。
Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution
- 通过可变分辨率编码器与动态压缩模块,支持1-16倍灵活压缩。
- 在视频等长序列上保持高精度,文档理解任务无压缩仍达高准确率。
- 适合需要处理多模态长视觉内容的研究者与开发者。
视觉数据形式多样,从几像素的小图标到数小时的视频不等。现有多模态大模型通常将输入统一为固定分辨率,导致对长短视觉内容处理效率低下。为此,我们提出Oryx,一种统一的多模态架构,用于图像、视频及多视角3D场景的时空理解。Oryx通过两项核心创新实现按需处理:1)预训练的OryxViT模型可将任意分辨率图像编码为大模型友好的视觉表征;2)动态压缩模块支持1×至16×的视觉标记压缩,按需启用。该设计使Oryx能以低分辨率和高压缩处理超长视觉上下文(如视频),同时在文档理解等任务中保持原生分辨率与无压缩的高识别精度。此外,通过增强数据构建与针对长上下文检索、空间感知数据的专项训练,Oryx在图像、视频、3D多模态理解方面均表现卓越。项目已开源:https://github.com/Oryx-mllm/Oryx。
原文摘要 · Abstract (English)
Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to a fixed resolution for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These design features enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。