用统一编码直接绑定语义与空间位置,让图像生成更高效精准。
UniVL: Unified Vision-Language Embedding for Spatially Grounded Contextual Image Generation

- 将文本指令直接渲染到空间掩码上,通过单一致编码器融合视觉与语义信息。
- 在47.7万张带掩码标注图像上,FID降至11,PSNR升至20,性能超越基线。
- 无需独立文本编码器,推理计算量减少52%,适合追求效率的生成应用。
我们提出空间对齐的上下文图像生成任务,重新定义条件生成范式。传统方法使用独立的视觉和语言编码器分别处理参考图与全局文本提示,而UniVL通过单一统一视觉输入,直接将语义绑定到空间位置,其中文本指令被渲染至空间掩码。该设计省去推理时独立的文本编码器。模型采用基于OCR预训练骨干的UniVL编码器,光学读取统一条件并生成融合视觉、语义意图与空间位置的单序列嵌入fVIL。两阶段流程先对齐UniVL与VAE嵌入空间,再完全基于UniVL嵌入条件化预训练扩散模型,彻底移除T5等独立文本编码器。尽管采用极简文本接口,仍取得显著性能提升:在自建的47.7万张掩码标注图像数据集UniVL-ImgGen上,图像质量优于文本提示基线,FID从14降至11,PSNR从16升至20;同时消除文本编码器,推理时钟周期减少44%,计算量降低52%。消融实验验证各组件贡献,为高效、空间对齐的图像生成提供统一条件范式。
原文摘要 · Abstract (English)
We introduce spatially grounded contextual image generation, a controllable image generation task that reframes the conditioning paradigm. Instead of supplying a reference image and a global text prompt through two separate encoders, one for vision and one for language, UniVL is trained to bind semantics to spatial locations directly from a single unified visual input, where the textual instruction is rendered onto the spatial mask. This removes the need for a standalone text encoder at inference time. The resulting model supports contextual image generation by following user-specified instructions about what should appear where, while substantially reducing computation. To address this task, we propose a framework in which the UniVL encoder, adapted from an optical-character-recognition-pretrained backbone, reads the unified condition optically and produces a UniVL embedding, fVIL, that fuses visual and semantic intent with spatial locations in a single token sequence. A two-stage pipeline first aligns UniVL with the VAE embedding space and then conditions a pretrained diffusion backbone entirely on UniVL embeddings, eliminating the standalone text encoder, such as T5. Although this reframing uses a deliberately minimal text interface, it yields strong empirical gains. On UniVL-ImgGen, a benchmark of 477K mask-annotated images that we construct for training and evaluation, UniVL improves image quality over text-prompted baselines, reducing FID from 14 to 11 and increasing PSNR from 16 to 20. It also eliminates the text encoder entirely, reducing inference TFLOPs by up to 52% and runtime by up to 44%. Additional ablation studies validate the contributions of the proposed components, paving the way for efficient, spatially grounded image generation with a unified conditioning paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。