用视觉提示中间引导,让图像生成更准更快。
Imagine Before You Draw: Visual Prompt Engineering for Image Generation

- 先生成视觉语义标记作蓝图,再据此绘图。
- 编辑时保留细节能力提升,PSNR达26.76。
- 适合需要高质量图像编辑的开发者使用。
在图像生成前引入视觉语义表示作为中间步骤,可降低文本与图像间的建模难度,从而提升生成质量。现有方法如X-Omni和BLIP3o-Next采用两阶段外部流水线:先用自回归模型生成语义标记,再交由独立扩散解码器生成图像。由于解码器无法同时访问原始输入与语义计划,该设计带来信息瓶颈,限制了编辑任务中的细节保留。内部架构如Transfusion、BAGEL和Show-o2虽通过单模型内跨模态交互避免了这一问题,但仍面临缺乏中间语义引导的文本到像素建模鸿沟。本文提出视觉提示工程(VPE),可无缝集成于此类内部框架。具体而言,模型首先自回归生成视觉语义标记(如SigLIP 2)作为“视觉提示”,捕捉语义布局,随后基于此计划生成完整图像标记。我们在类别条件生成、文生图生成及图像编辑任务中验证了VPE,涵盖多种标记类型与模型架构。结果表明,VPE能加速收敛、提升质量上限,并通过内部集成实现显著更好的编辑保留能力(PSNR: 26.76 vs. 19.92),在同等参数量下优于外部方案,同时保持良好的编辑响应性。
原文摘要 · Abstract (English)
Incorporating visual semantic representations as an intermediate step before image generation can reduce the modeling difficulty between text and images, thereby improving generation quality. Recent works such as X-Omni and BLIP3o-Next have explored this direction, but they typically use a two-stage external pipeline: a separate autoregressive model first generates semantic tokens, which are then fed as conditioning to an independent diffusion decoder. Since the decoder cannot jointly access the original input and the semantic plan, this design introduces an information bottleneck that limits detail preservation in downstream tasks such as editing. Internal architectures such as Transfusion, BAGEL, and Show-o2 avoid this bottleneck by enabling cross-modal interaction within a single model, but they still face the difficult text-to-pixel modeling gap without intermediate semantic guidance. We propose Visual Prompt Engineering (VPE), which can be seamlessly integrated into such internal frameworks. Specifically, the model first autoregressively generates visual semantic tokens (e.g., SigLIP 2) as "visual prompts" that capture the semantic layout, then generates the full image tokens conditioned on this plan. We validate VPE across class-conditional generation, text-to-image generation, and image editing, covering various token types and model architectures. Results show that VPE can accelerate convergence, raise quality ceilings, and through internal integration, achieve substantially better editing preservation (PSNR: 26.76 vs. 19.92) than external alternatives of the same parameter scale, while maintaining competitive editing responsiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。