arXiv:2511.16917cs.CV2025-11被引 4

用纯视觉空间统一图文理解与生成,实现端到端的跨模态转换。

UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation

论文配图:UniModel: A Visual-Only Framework for Unified Multimodal Understanding and Generation
图 1 · 摘自论文原文
  • 将文本转为画出的文字图像,所有输入输出统一为像素,实现视觉原生建模。
  • 在图像到文字和文字到图像任务中均达到强对齐效果,支持循环一致的图文互推。
  • 适合追求统一框架、简化多模态系统的研究者或开发者使用。

我们提出UniModel,一个统一的生成模型,在单一像素到像素的扩散框架内同时支持视觉理解和视觉生成。目标是在模型、任务和表征三个维度实现统一。在表征层面,通过将文本和图像映射到共享的视觉空间,消除模态差异:文本提示被渲染为干净画布上的绘制文字图像,所有输入输出仅作为RGB像素处理。这实现了完全视觉原生的多模态学习。在任务层面,广泛的视觉语言问题被转化为该视觉空间中的像素到像素变换。对于理解任务,模型接收RGB图像并生成编码语义预测的绘制文字图像;对于生成任务,绘制文字图像作为视觉条件引导真实且语义对齐的图像合成。图像描述和文生图因此成为同一底层视觉翻译过程的不同方向。在模型层面,我们采用在像素空间中训练的统一扩散变压器,通过修正流学习。共享主干网络联合学习自然图像与绘制文字图像之间的双向映射,并通过轻量级任务嵌入指定期望方向。在文生图和图生文理解任务上的实验表明,具有强跨模态对齐和涌现的可控性,例如支持循环一致的图像-标题-图像闭环。初步探索表明,在单一视觉空间中统一模型、任务与表征是一种有前景的通用多模态智能范式。

原文摘要 · Abstract (English)

We present UniModel, a unified generative model that jointly supports visual understanding and visual generation within a single pixel-to-pixel diffusion framework. Our goal is to achieve unification along three axes: the model, the tasks, and the representations. At the representation level, we eliminate modality discrepancies by mapping both text and images into a shared visual space: textual prompts are rendered as painted text images on a clean canvas, and all inputs and outputs are treated purely as RGB pixels. This yields a fully vision-native formulation of multimodal learning. At the task level, a broad range of vision-language problems are cast as pixel-to-pixel transformations in this visual space. For understanding tasks, the model takes an RGB image and produces a painted text image that visually encodes the semantic prediction. For generation tasks, painted text images serve as visual conditions that guide realistic and semantically aligned image synthesis. Captioning and text-to-image generation thus become different directions of the same underlying visual translation process. At the model level, we instantiate a single Unified Diffusion Transformer trained with rectified flow in pixel space. A shared backbone jointly learns bidirectional mappings between natural images and painted text images, with lightweight task embeddings to specify the desired direction. Experiments on text-to-image synthesis and image-to-text understanding demonstrate strong cross-modal alignment and emergent controllability such as cycle-consistent image-caption-image loops. Our initial exploration suggests that unifying model, tasks, and representations in a single visual space is a promising paradigm for general-purpose multimodal intelligence.

多模态扩散模型统一框架视觉生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。