arXiv:2601.04706cs.CV2026-01被引 1

用理解模型提升图像生成细节,让画得更真更细。

Forge-and-Quench: Enhancing Image Generation for Higher Fidelity in Unified Multimodal Models

  • 用理解模型提炼更强文本指令,生成更逼真的图像。
  • 通过桥接特征注入视觉引导信号,显著提升图像质量。
  • 可适配多种模型,训练成本低,适合快速部署。

将图像生成与理解整合到统一框架是多模态领域的关键目标。然而,如何有效利用理解能力来增强生成仍不明确。本文提出一种新视角:利用理解模型提升生成图像的保真度和细节丰富度。为此,我们设计了 Forge-and-Quench 框架,在生成过程中,先由多模态大模型(MLLM)对完整对话上下文进行推理,生成优化后的文本指令;再通过新型桥接适配器(Bridge Adapter),将该指令映射为虚拟视觉表征——桥接特征(Bridge Feature),作为理解模型与生成过程之间的关键桥梁。该特征被注入到文本到图像(T2I)主干网络中,与优化后的文本指令共同作为输入。实验验证了桥接特征与适配器的设计有效性。该框架具备极强可扩展性,能高效迁移至不同 MLLM 与 T2I 模型,大幅降低训练开销,且不损害 MLLM 的多模态理解能力。结果表明,该方法在多个模型上均显著提升了图像保真度与细节表现,同时保持指令遵循准确率,并增强世界知识应用能力。代码与模型已开源。

原文摘要 · Abstract (English)

Integrating image generation and understanding into a single framework has become a pivotal goal in the multimodal domain. However, how understanding can effectively assist generation has not been fully explored. Unlike previous works that focus on leveraging reasoning abilities and world knowledge from understanding models, this paper introduces a novel perspective: leveraging understanding to enhance the fidelity and detail richness of generated images. To this end, we propose Forge-and-Quench, a new unified framework that puts this principle into practice. In the generation process of our framework, an MLLM first reasons over the entire conversational context, including text instructions, to produce an enhanced text instruction. This refined instruction is then mapped to a virtual visual representation, termed the Bridge Feature, via a novel Bridge Adapter. This feature acts as a crucial link, forging insights from the understanding model to quench and refine the generation process. It is subsequently injected into the T2I backbone as a visual guidance signal, alongside the enhanced text instruction that replaces the original input. To validate this paradigm, we conduct comprehensive studies on the design of the Bridge Feature and Bridge Adapter. Our framework demonstrates exceptional extensibility and flexibility, enabling efficient migration across different MLLM and T2I models with significant savings in training overhead, all without compromising the MLLM's inherent multimodal understanding capabilities. Experiments show that Forge-and-Quench significantly improves image fidelity and detail across multiple models, while also maintaining instruction-following accuracy and enhancing world knowledge application. Models and codes are available at https://github.com/YanbingZeng/Forge-and-Quench.

图像生成多模态理解增强桥接特征

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。