无需预处理，直接生成图文和表格等多模态数据。
Diffuse Everything: Multimodal Diffusion Models on Arbitrary State Spaces
- 在任意状态空间构建多模态扩散模型，避免依赖编码器。
- 采用分治噪声调度，单模型实现无条件与条件生成。
- 适合数据少、需联合生成多种类型数据的场景。
扩散模型在图像、视频、文本等单模态生成任务中表现卓越，但在多模态联合生成方面仍处于探索阶段。现有方法高度依赖外部预处理流程（如分词器、变分自编码器）将不同数据表示统一为单一模态格式，这对编码器和解码器的精度要求极高，尤其在数据有限的应用中难以满足。为此，我们提出一种新型框架，可在任意状态空间上构建多模态扩散模型，实现跨模态耦合数据的原生生成。通过引入针对各模态独立设计的噪声调度策略，使单个模型同时支持无条件生成与模态条件生成。我们在文本-图像生成和混合类型表格数据合成任务上进行实证验证，结果表明该方法性能具有竞争力。
原文摘要 · Abstract (English)
Diffusion models have demonstrated remarkable performance in generating unimodal data across various tasks, including image, video, and text generation. On the contrary, the joint generation of multimodal data through diffusion models is still in the early stages of exploration. Existing approaches heavily rely on external preprocessing protocols, such as tokenizers and variational autoencoders, to harmonize varied data representations into a unified, unimodal format. This process heavily demands the high accuracy of encoders and decoders, which can be problematic for applications with limited data. To lift this restriction, we propose a novel framework for building multimodal diffusion models on arbitrary state spaces, enabling native generation of coupled data across different modalities. By introducing an innovative decoupled noise schedule for each modality, we enable both unconditional and modality-conditioned generation within a single model simultaneously. We empirically validate our approach for text-image generation and mixed-type tabular data synthesis, demonstrating that it achieves competitive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。