OmniFlow实现跨模态任意生成，支持文本、图像、音频互转。
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
- 扩展时序流框架至多模态，引入灵活控制模态对齐的新引导机制。
- 基于Stable Diffusion 3的MMDiT架构扩展，支持音频与文本生成。
- 可分步预训练模块，便于高效微调，适合多模态生成研究者。
我们提出OmniFlow，一种新型生成模型，用于实现任意模态间的生成任务，如文本到图像、文本到音频以及音频到图像合成。OmniFlow将文本到图像模型中使用的修正流（Rectified Flow, RF）框架拓展至多模态联合分布建模，显著优于现有任意模态生成模型，在文本到图像和文本到音频等任务上表现优异。本工作有三项关键贡献：第一，将修正流框架推广至多模态场景，并提出一种新颖的引导机制，使用户可灵活控制生成结果中不同模态间的对齐关系；第二，提出一种新架构，扩展了Stable Diffusion 3中的文本到图像MMDiT结构，支持音频与文本生成，其扩展模块可独立高效预训练并合并至原始文本到图像MMDiT进行微调；第三，系统研究了适用于大规模音频与文本生成的修正流变压器设计选择，为跨模态性能优化提供重要洞见。代码将发布于 https://github.com/jacklishufan/OmniFlows。
原文摘要 · Abstract (English)
We introduce OmniFlow, a novel generative model designed for any-to-any generation tasks such as text-to-image, text-to-audio, and audio-to-image synthesis. OmniFlow advances the rectified flow (RF) framework used in text-to-image models to handle the joint distribution of multiple modalities. It outperforms previous any-to-any models on a wide range of tasks, such as text-to-image and text-to-audio synthesis. Our work offers three key contributions: First, we extend RF to a multi-modal setting and introduce a novel guidance mechanism, enabling users to flexibly control the alignment between different modalities in the generated outputs. Second, we propose a novel architecture that extends the text-to-image MMDiT architecture of Stable Diffusion 3 and enables audio and text generation. The extended modules can be efficiently pretrained individually and merged with the vanilla text-to-image MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design choices of rectified flow transformers for large-scale audio and text generation, providing valuable insights into optimizing performance across diverse modalities. The Code will be available at https://github.com/jacklishufan/OmniFlows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。