arXiv:2503.10772cs.CV2025-03ICCV被引 34

用统一1D令牌流实现图文无缝生成,效率更高且无需复杂条件机制。

FlowTok: Flowing Seamlessly Across Text and Image Tokens

  • 将图像压缩为1D令牌,与文本共享同一潜在空间进行流匹配。
  • 256分辨率下潜空间缩小3.3倍,采样速度显著提升。
  • 架构简洁高效,适合资源受限场景下的跨模态生成任务。

跨模态生成的核心在于融合不同模态。传统方法将文本作为条件信号,逐步引导去噪过程从高斯噪声生成目标图像。本文提出更简洁的范式:直接通过流匹配在文本与图像模态间演化。为此,需将两者映射至共享潜在空间,但文本为高语义1D标记,图像具空间冗余性且为2D嵌入,存在本质差异。为此,我们提出FlowTok,通过将图像编码为紧凑1D令牌表示,实现图文无缝流动。相比以往方法,该设计在256分辨率下使潜空间大小减少3.3倍,无需复杂条件机制或噪声调度。此外,该框架自然扩展至图像到文本生成。基于紧凑1D令牌的轻量架构,FlowTok内存效率高,训练资源需求少,采样速度快,性能媲美当前最优模型。代码已公开于https://github.com/TACJu/FlowTok。

原文摘要 · Abstract (English)

Bridging different modalities lies at the heart of cross-modality generation. While conventional approaches treat the text modality as a conditioning signal that gradually guides the denoising process from Gaussian noise to the target image modality, we explore a much simpler paradigm-directly evolving between text and image modalities through flow matching. This requires projecting both modalities into a shared latent space, which poses a significant challenge due to their inherently different representations: text is highly semantic and encoded as 1D tokens, whereas images are spatially redundant and represented as 2D latent embeddings. To address this, we introduce FlowTok, a minimal framework that seamlessly flows across text and images by encoding images into a compact 1D token representation. Compared to prior methods, this design reduces the latent space size by 3.3x at an image resolution of 256, eliminating the need for complex conditioning mechanisms or noise scheduling. Moreover, FlowTok naturally extends to image-to-text generation under the same formulation. With its streamlined architecture centered around compact 1D tokens, FlowTok is highly memory-efficient, requires significantly fewer training resources, and achieves much faster sampling speeds-all while delivering performance comparable to state-of-the-art models. Code is available at https://github.com/TACJu/FlowTok.

跨模态生成流匹配1D令牌高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。