用能量模型提升多模态生成的先验表达能力
Learning Multimodal Latent Generative Models with Energy-Based Prior
- 将能量模型作为多模态生成的先验，增强信息捕捉能力
- 联合训练框架使生成结果在跨模态上更连贯
- 适合关注多模态生成与先验设计的研究者
多模态生成模型近年来受到广泛关注，因其能够学习多种模态间的表示，提升联合生成与跨模态生成的一致性。然而，大多数现有方法使用标准高斯或拉普拉斯分布作为先验，这些单峰且信息量有限的分布难以捕捉多种数据类型中的复杂多样性。能量模型（EBM）以其在各类任务中的表达力和灵活性著称，但在多模态生成模型中尚未得到充分探索。本文提出一种新框架，将多模态潜在生成模型与能量模型相结合，通过变分方案实现两者的联合训练。该方法构建了更具表达力和信息量的先验，能更好捕捉多模态间的信息。实验验证了所提模型的有效性，展示了其在生成一致性方面的显著优势。
原文摘要 · Abstract (English)
Multimodal generative models have recently gained significant attention for their ability to learn representations across various modalities, enhancing joint and cross-generation coherence. However, most existing works use standard Gaussian or Laplacian distributions as priors, which may struggle to capture the diverse information inherent in multiple data types due to their unimodal and less informative nature. Energy-based models (EBMs), known for their expressiveness and flexibility across various tasks, have yet to be thoroughly explored in the context of multimodal generative models. In this paper, we propose a novel framework that integrates the multimodal latent generative model with the EBM. Both models can be trained jointly through a variational scheme. This approach results in a more expressive and informative prior, better-capturing of information across multiple modalities. Our experiments validate the proposed model, demonstrating its superior generation coherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。