arXiv:2605.20316cs.CVcs.AI2026-05

用轻量适配器让文生图模型同时支持图生文,性能暴涨。

FullFlow: Upgrading Text-to-Image Flow Matching Models for Bidirectional Vision--Language Generation

  • 仅训练低秩适配器和轻量文本头,不改原模型架构
  • 文生图FID降为31.6(原62.7),图生文CIDEr升至99.4
  • 仅调5%参数,24小时内完成训练,显存降低超一半

当前文生图扩散模型仅支持单向生成,现有统一视觉-语言模型需大规模联合预训练或重训文本路径,放弃原有图像先验。我们提出FullFlow,通过仅训练LoRA适配器和轻量文本头,将预训练的修正流文生图模型升级为双向生成器。保留图像连续流表示,引入离散文本插入过程,分离图像与文本时间步,使推理变为二维生成空间中的轨迹选择,实现文→图、图→文、联合采样及部分文本预测。在稳定扩散3(SD3)上,相同可训练参数量与LoRA秩下,相比此前最优方案(双扩散),文→图FID从62.7降至31.6,图→文CIDEr从2.0升至99.4,峰值显存由约84GB降至约38GB,吞吐提升约8倍,仅训练约5%主干参数,在两块RTX A5000上24小时内完成。该方法同样适用于FLUX.1-dev,并支持下游视觉问答任务。结果表明,无需全模态预训练即可从文生图模型中解锁强双向能力。

原文摘要 · Abstract (English)

Modern text-to-image diffusion models encode rich visual priors, but expose them only through one-way text-conditioned generation. Existing unified vision--language models derived from them recover bidirectional capability through large-scale joint pretraining or substantial retraining of the text pathway, discarding the strong image prior the text-to-image backbone already encodes. We introduce \emph{FullFlow}, a parameter-efficient recipe that upgrades a pretrained rectified-flow text-to-image model into a bidirectional vision--language generator by training only LoRA adapters and lightweight text heads. FullFlow keeps images in their native continuous flow and adds a discrete insertion process for text. Separate image and text timesteps turn inference into trajectory selection in a two-dimensional generative space, enabling text$\rightarrow$image, image$\rightarrow$text, joint sampling, and partial-text prediction with a single backbone. On Stable Diffusion 3 (SD3) under an identical trainable-parameter count and matched LoRA rank, FullFlow improves text$\rightarrow$image FID from $62.7$ to $31.6$ and image$\rightarrow$text CIDEr from $2.0$ to $99.4$ over a LoRA equivalent following the previous SOTA formulation (Dual Diffusion) at matched wall-clock training time, while reducing peak VRAM from ${\sim}84$\,GB to ${\sim}38$\,GB and raising throughput by ${\sim}8\times$ on two RTX A5000 GPUs in under 24 hours, training only ${\sim}5\%$ of the backbone parameters. The same recipe transfers to FLUX.1-dev and supports downstream VQA through partial-text generation. These results show that strong bidirectional vision--language capability can be unlocked from pretrained text-to-image flow models without full multimodal pretraining.

文生图双向生成LoRA高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。