用统一生成框架让模型用自然语言搞定各种视觉任务。
Vision as Unified Multimodal Generation

- 把不同视觉任务都转成文本和图像生成问题,不改结构
- 单模型覆盖检测、分割、深度估计等10+任务,性能媲美专用模型
- 适合想用通用模型做多类视觉任务的研究者和开发者
本文将计算机视觉重新定义为统一的多模态生成任务,将异构的视觉任务表达在统一多模态模型的原生文本与图像生成空间中,无需任务特定架构。SenseNova-Vision通过自然语言指令和可选视觉提示指定任务、目标区域或视角,以文本、图像或图文混合输出响应。为支持大规模训练,我们将多样化的计算机视觉标注转换为适配生成空间的指令-响应样本,构建了涵盖文本、图像及混合目标的SenseNova-Vision语料库。基于预训练的统一多模态模型,该系统主要在该语料库上训练,辅以多模态数据保持能力,无需任务特定预测头或结构修改。模型覆盖检测、OCR、关键点估计、分割、深度估计、表面法向预测、点图、相机位姿估计等广泛任务,并支持由语言定义的类别、颜色、区域等组合线索。实验表明,单一统一模型在结构化视觉理解、密集几何预测、分割和多视图视觉几何任务上达到领先专用系统的性能。结果表明,统一多模态生成是将视觉能力集成到通用基础模型的可扩展路径。模型与语料库已公开。
原文摘要 · Abstract (English)
We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。