用流模型统一视觉与语言模型词汇，实现基于大模型的图像自回归生成。
V2Flow: Unifying Visual Tokenization and Large Language Model Vocabularies for Autoregressive Image Generation
- 将视觉分词建模为流匹配问题，使图像分布与语言模型词汇空间对齐。
- 在ImageNet上重建质量达3.17的PSNR，优于主流VQ方法。
- 可直接在现有大语言模型上进行图像自回归生成，适合多模态研究者。
我们提出V2Flow，一种新型分词器，能生成高保真重建的离散视觉标记，并确保其结构和潜在分布与大型语言模型（LLM）词汇空间一致。通过这种紧密的视觉-词汇耦合，V2Flow使得在现有LLM之上实现自回归视觉生成成为可能。我们的方法将视觉分词建模为流匹配问题，目标是学习从标准正态先验到连续图像分布的映射，条件于嵌入在LLM词汇空间中的标记序列。V2Flow的有效性源于两个核心设计：首先，我们提出视觉词汇重采样器，将视觉数据压缩为紧凑的标记序列，每个标记以软类别分布表示，从而实现与现有LLM的无缝集成；其次，我们设计了带掩码的自回归修正流解码器，使用掩码Transformer编码器-解码器精炼视觉标记为上下文丰富的嵌入，再由专用速度场实现精确重建。此外，引入自回归修正流采样策略，支持灵活序列长度的同时保持竞争性重建质量。大量实验表明，V2Flow优于主流基于VQ的分词器，并成功实现了在现有大语言模型上的自回归视觉生成。
原文摘要 · Abstract (English)
We propose V2Flow, a novel tokenizer that produces discrete visual tokens capable of high-fidelity reconstruction, while ensuring structural and latent distribution alignment with the vocabulary space of large language models (LLMs). Leveraging this tight visual-vocabulary coupling, V2Flow enables autoregressive visual generation on top of existing LLMs. Our approach formulates visual tokenization as a flow-matching problem, aiming to learn a mapping from a standard normal prior to the continuous image distribution, conditioned on token sequences embedded within the LLMs vocabulary space. The effectiveness of V2Flow stems from two core designs. First, we propose a Visual Vocabulary resampler, which compresses visual data into compact token sequences, with each represented as a soft categorical distribution over LLM's vocabulary. This allows seamless integration of visual tokens into existing LLMs for autoregressive visual generation. Second, we present a masked autoregressive Rectified-Flow decoder, employing a masked transformer encoder-decoder to refine visual tokens into contextually enriched embeddings. These embeddings then condition a dedicated velocity field for precise reconstruction. Additionally, an autoregressive rectified-flow sampling strategy is incorporated, ensuring flexible sequence lengths while preserving competitive reconstruction quality. Extensive experiments show that V2Flow outperforms mainstream VQ-based tokenizers and facilitates autoregressive visual generation on top of existing. https://github.com/zhangguiwei610/V2Flow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。