MUNI实现跨模态任意生成,统一条件与无条件采样。
MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

- 用共享随机隐空间统一多模态生成流程,端到端训练。
- 在图像-文本-音频数据集上,无条件生成一致性优于基线。
- 适合需要跨模态协同生成的研究者和开发者。
我们提出MUNI,一种用于任意模态间生成的端到端多模态潜在扩散框架,通过共享随机隐空间统一子集条件交叉模态生成与无条件联合采样。现有模型多基于大语言模型,限制了模态特异性生成器的应用,并需文本配对数据训练。近期扩散与流模型的扩展虽走不同路径,但仍依赖文本对齐嵌入、全配对训练或维度匹配确定性映射。MUNI基于两项互补贡献:其一为架构创新,将潜在扩散扩展至多模态任意生成,不再分两阶段预计算冻结隐空间再拟合先验,而是联合训练各模态编码器、表达力强的解码器与单一流式先验;其二为训练目标设计,发现标准多模态变分推断聚合规则在结合学习先验与表达解码器后不足。理想的共享隐空间需同时满足生成模态间的连贯性、子集隐变量的预测充分性及隐含内容最小性。我们提出路由训练目标,其结构选择使隐空间契合上述标准,并在可实现设定下具备最小充分性特征。在PolyMNIST-Quadrant-Labels与大规模图像-文本-音频基准上的实验表明,MUNI在条件生成上达到或超越最强基线,且在无条件连贯性上取得最大优势。
原文摘要 · Abstract (English)
We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent. Existing multimodal generative models are largely LLM-based, which limits leveraging modality-specific generators and requires text-paired data for training. Recent diffusion- and flow-based any-to-any extensions take a different direction but still rely on text-aligned embeddings, fully-paired training, or matched-dimensionality deterministic mappings. MUNI rests on two complementary contributions, one architectural and one in the training objective. First, we extend latent diffusion to multimodal any-to-any generation end-to-end: instead of the standard two-stage recipe that precomputes a frozen latent space and then fits a prior over it, MUNI jointly trains modality-specific encoders, expressive decoders, and a single shared flow-based prior under one objective. Second, we identify that the standard aggregation rules of multimodal variational inference are insufficient once coupled with a learned prior and expressive decoders. A suitable shared latent must simultaneously satisfy coherence across generated modalities, predictive sufficiency of subset latents, and minimality of the latent content. We propose a routed training objective whose structural choices align the latent with these criteria and admit a minimal-sufficiency characterization in the realizable setting. Experiments on PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark show MUNI matching or exceeding the strongest baselines on conditional generation while opening its largest margins on unconditional coherence. Project page: https://muni-proj.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。