arXiv:2606.04264cs.CV2026-06

用扩散模型实现图文混排生成,让文字自然融入图像绘制过程。

UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation

论文配图:UniCanvas: A Diffusion-base Unified Model for Text-in-Image Joint Generation
图 1 · 摘自论文原文
  • 将语言编码为图像内的视觉模式,统一在像素画布上生成图文内容。
  • 相比以往模型,显著提升图文联合生成的连贯性与质量。
  • 适合研究多模态生成、图像编辑与创意设计的开发者。

近年来,统一视觉-语言模型在单架构中实现了多模态理解与生成的突破。尽管自回归模型能跨模态推理,但难以生成高质量图像;而扩散模型虽能生成逼真视觉内容,却难以生成连贯文本,导致统一模型难以兼顾两者。最新进展表明,语言可有效嵌入视觉表征,使模型能直接从图像中理解语义。为此,我们提出UniCanvas,首个基于扩散模型的统一多模态生成框架,通过文本-图像联合生成实现无缝融合。扩散模型天然具备在共享像素画布上的变换能力,可视为视觉变化的世界模型。模型不输出离散文本标记,而是将语言表示为图像内部的视觉模式,利用其固有的多模态嵌入空间。该设计使模型在图像合成过程中“自然绘制”文字,实现端到端的多模态生成。实验表明,UniCanvas在性能上优于现有统一模型,验证了扩散模型在文本-图像生成中的潜力,为统一多模态生成提供新范式。

原文摘要 · Abstract (English)

Recent years have seen remarkable progress in unified vision-language models handling both multimodal understanding and generation within a single architecture. While autoregressive VLMs can reason across modalities, they fail to generate high-quality images. In contrast, diffusion models produce photorealistic visuals yet struggle to generate coherent text, making it challenging to develop a single unified model that can seamlessly handle both visual and text generation. Recent advances suggest that language can be effectively embedded within visual representations, allowing models to reason about textual semantics directly from images. To this end, we propose UniCanvas, a first attempt that unifies diffusion models to generate interleaved multimodal contents through text-in-image generation. Diffusion models naturally capture transformations on a shared pixel canvas, which can be viewed as world models of visual change. Instead of producing discrete text tokens, the model learns to represent language as visual patterns inside images, leveraging its inherent multimodal embedding space. This design allows the model to "draw" text naturally within a single pixel canvas during image synthesis, achieving seamless multimodal generation. Experiments demonstrate that UniCanvas improves performance over previous unified models, positioning text-in-image generation with diffusion models as a promising unified multimodal generation paradigm.

扩散模型图文生成多模态图像生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。